https://linen.dev logo
Join Slack
Powered by
Hi all, I'm just getting into Meltano right now as...
# getting-started
e
Hi all, I'm just getting into Meltano right now as I evaluate it and I'm a bit confused on the schema relationship between 
streams.py
and 
tap.py
specifically in the tap template. I read through the "Getting Started" template and the Code Samples on the site but still not really understanding it. From my understanding is that the schema defined in the 
streams.py
is the JSON that is being sent from the source and 
tap.py
is the target? But then what happens when we load the cookiecutter target template? I thought taps was used for bringing in data from the source and target was used to send data to a target. For example this is the flow I'm trying to create: API source (using a tap) -> S3 bucket -> Postgres (I assume using a target?)
t
While I haven't developed a tap myself (yet): based on the SDK docs I would expect that tap.py is for managing configuration and streams.py is for communicating with your source (the API). Neither have anything to do with the target, that's for sure. Q: Are you sending the data to S3 first so you can re-use it elsewhere? If you just want to get the data from the API to Postgres your tap can (and should) just produce the JSON containing the data; the target will consume that and load it to PG itself.
a
Hi, @edward_chen. While developing in the SDK: • The Tap class handles high-level orchestration, invocation, and CLI functions. This is in 
tap.py
. • The Stream class(es) handle API-specific functions, such as defining the schema of the stream, primary keys (key properties), and parent-child relationships. These generally live in 
streams.py
(although some developers break off into smaller files) and sometimes one or more abstract base classes will live in 
client.py
. Does this help?
The target is fully separate from the tap - whatever schema you define for your streams, and whichever primary key or replication keys you define, those will be passed along for the target to leverage in creating the receiving data structures.
e
Ah got it, thank you that helps tremendously in understanding the relationship. I noticed that some code examples didn't have a 
client.py
so that makes sense if it's used as more of an abstract layer to organize the code. So if the code is simple enough, you could put the classes in 
streams.py
So in my case, I would want a tap that pulls from my API source and a S3 target right? If I'm understanding the Tap/Target relationship
a
So if the code is simple enough, you could put the classes in 
streams.py
Yep!
So in my case, I would want a tap that pulls from my API source and a S3 target right? If I'm understanding the Tap/Target relationship
That's right!
There's one wrinkle though I want to call out... the two-hop process, of using an S3 target and then also wanting to post to Postgres might be a bit tricky.
It's probably easier to write a target that lands data in both postgres and S3 than to have those as separate links in the chain. The reason is that when you scan metadata in S3, you likely will have less structure in the catalog than you had originally in Postgres. Most "big data" targets like Redshift and Snowflake actually write data to S3 storage internally as the first step. This is transparent to the users of the target, but some variants of those allow you to retain the S3 artifacts after loading into Redshift/Snowflake/etc., rather than cleaining up the file resources afterwards.
e
Ah got it, when you say it might be a bit tricky, what do you anticipate could go wrong or be hard to configure? Why I'm thinking of loading to S3 first is because I'm still evaluating the "big data" targets so I don't know if I will end up with Snowflake or Postgres so the intention is to load everything into S3 first and then I can pull from the S3 buckets for each of the "big data" targets.
Also sorry, while rereading your messages I'm still not 100% sure on what values I would put in 
tap.py
for 
config_jsonschema = PropertiesList(...)
. What properties are supposed to go there? I understand that in 
streams.py
, it's the schema of the incoming "raw" data.
a
What's your data source?
e
I'm looking at bringing in data from Snyk, a cybersecurity tool that scans tools. So the data schema is something like this:
Copy code
{
  "type": "object",
  "properties": {
    "id": {
      "type": "string"
    },
    "severity": {
      "type": "string"
    },
    "dependencies": {
      "type": "array"
    },
    "projects": {
      "type": "array"
    }
  }
}
a
Got it, yeah makes sense.
I'm still not 100% sure on what values I would put in 
tap.py
for 
config_jsonschema = PropertiesList(...)
.
In tap.py, for config_jsonschema, you are defining the config flags for the user. So, if you need auth or a url endpoint, or other settings that the tap will use as control inputs, those would go into 
config_jsonschema
. This is similar in specification with the 
schema
properties of 
Streams
but the function is very different. In the Tap, you are defining config, and in the Stream, you are defining the schema of the stream's data. Does that help? 🙂
e
Appreciate it, I'll keep rereading that to see if I can understand that. I guess I'm still a little uncertain on the config, what types of config do you mean? Sorry about that, I feel like it's obvious but I can't seem to grasp it. Could you provide an example? (no code necessary if it makes it quicker)
By configs, do you mean settings like user configurations? So if for example an API source has different permissions, I would put something like:
Copy code
config_jsonschema = PropertiesList(Property("permissions", StringType))
assuming the API source requires something like 
{"permissions": "read-only"}
?
a
No worries at all. Maybe this is helpful: https://hub.meltano.com/taps/mysql#settings
For that tap, the needed settings are 
host
, 
port
, 
user
, 
password
, etc.
e
AHH okay perfect, thank you that does clear things up
a
Great!
Open in Slack
PreviousNext
Powered by