https://linen.dev logo
Join Slack
Powered by
1. I am having issues setting up multiple pipeline...
# announcements
z
2. I am having issues setting up multiple pipelines with the same tap-facebook. .env file contains default 
TAP_FACEBOOK_ACCOUNT_ID=''
and then meltano.yml contains
Copy code
extractors:
  - name: tap-facebook
    pip_url: git+<https://gitlab.com/meltano/tap-facebook.git|https://gitlab.com/meltano/tap-facebook.git>
    config:
      start_date: '2020-10-02'
      insights_buffer_days: 7,
    settings:
    - name: account_id
      env: TAP_FACEBOOK_ACCOUNT_ID
...
schedules:
- name: facebook-linson-gds
  extractor: tap-facebook
  loader: target-bigquery
  transform: skip
  interval: '@hourly'
  start_date: 2020-10-10 00:00:00
  env:
    TAP_FACEBOOK_ACCOUNT_ID: 296651825
Tried to have the ACCOUNT_ID with single quotes, but it is still not working. When running the scheduler it says that no account_id is found.
d
@zlatan When you run 
TAP_FACEBOOK_ACCOUNT_ID=296651825 meltano elt tap-facebook target-bigquery --job_id=facebook-linson-gds --dump=extractor-config
, is 
account_id
populated in the resulting config JSON?
z
sudo docker run --interactive -v $(pwd):/projects -w /projects TAP_FACEBOOK_ACCOUNT_ID=296651825 meltano/meltano elt tap-facebook target-bigquery --job_id=facebook-linson-gds1 --dump=extractor-config
Returns 
docker: invalid reference format: repository name must be lowercase.
d
Try this:
Copy code
sudo docker run --interactive -v $(pwd):/projects -w /projects -e TAP_FACEBOOK_ACCOUNT_ID=296651825 meltano/meltano elt tap-facebook target-bigquery --job_id=facebook-linson-gds1 --dump=extractor-config
z
This one worked
It has a populated account_id
d
@zlatan But apparently not when you use 
meltano invoke airflow scheduler
? Do you see any ``Considering schedule ...`` log messages in the output? Do you see 
"env": {"TAP_FACEBOOK_ACCOUNT_ID": 296651825}
in there?
z
Yes, many considering schedule and the env seems to be there
But then it doesn't actually run them
Or runs with no results
d
Are you still seeing the "no account_id found" error messages?
If there are no runs, that might be because an earlier run already succeeded and the time interval hasn't expired yet for the new run to start
z
If I run the schedule manually from Meltano UI, does it take the env from the meltano.yml > schedules ?
message has been deleted
d
Not currently, that's a known bug: https://gitlab.com/meltano/meltano/-/issues/2379
z
Each failed
d
Right. 
meltano invoke airflow schedule
still wouldn't run the pipeline again until the next interval tick. If https://meltano.slack.com/archives/CFG3C3C66/p1602610685178100?thread_ts=1602607187.174200&amp;cid=CFG3C3C66 worked, it should also work when Airflow runs the command instead, but it would be awesome if we could solve https://gitlab.com/meltano/meltano/-/issues/2379, and add 
meltano schedule run <name>
to make it easier to debug these schedules: https://gitlab.com/meltano/meltano/-/issues/2227
z
Does it take the correct env when run from airflow UI "Trigger Dag"?
d
Yeah it should
Does that work?
z
It fails
message has been deleted
It seems that the env for account id is there, but it is still not completing it
It seems like a very elegant solution in comparison with multiple tap-facebook installs
d
Can you share the failure error message?
z
Copy code
[2020-10-13 17:05:15,144] {bash_operator.py:146} INFO - Running command: cd /projects; .meltano/run/bin elt tap-facebook target-bigquery --transform=skip --job_id=facebook-linson-gds1
[2020-10-13 17:05:15,145] {taskinstance.py:1150} ERROR - expected str, bytes or os.PathLike object, not int
Traceback (most recent call last):
  File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/projects/.meltano/orchestrators/airflow/venv/lib/python3.6/site-packages/airflow/operators/bash_operator.py", line 151, in execute
    preexec_fn=pre_exec)
  File "/usr/local/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/local/lib/python3.6/subprocess.py", line 1275, in _execute_child
    env_list.append(k + b'=' + os.fsencode(v))
  File "/usr/local/lib/python3.6/os.py", line 800, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not int
[2020-10-13 17:05:15,148] {taskinstance.py:1194} INFO - Marking task as FAILED. dag_id=meltano_facebook-linson-gds1, task_id=extract_load, execution_date=20201013T160000, start_date=20201013T170514, end_date=20201013T170515
[2020-10-13 17:05:19,875] {local_task_job.py:102} INFO - Task exited with return code 1
This is the error message in the Airflow Log
d
@zlatan Does the error change if you wrap the ID in quotes in 
meltano.yml
?
z
It seems that it works. I'll try with multiple schedules. Do I have to edit max concurrent connections if I want to run 20 simultaneous ETLs, or does it handle them by default?
d
On your target database, you mean? It'll depend on how target-bigquery is implemented. I expect that it will fail when it tries to connect and the DB says it can't handle another connection, but it may automatically retry after some time. I suggest incrementing the max concurrent connections if you're actually expecting that many pipelines to run in parallel.
z
Got it, thanks for the input 🙂
11 Views
Open in Slack
PreviousNext
Powered by