https://linen.dev logo
Join Slack
Powered by
for example: where do the environment variable and...
# plugins-general
t
for example: where do the environment variable and default fit in this .yml file example? could i get guidance for this?
d
If you don't set a value for 
database
at all, the default value 
warehouse
will be used, and tap-sqlite will create a new SQLite database named 
warehouse.db
in your project directory if it doesn't exist yet.
If you'd like to use a different database file name, you can set the 
database
setting in 
meltano.yml
to a different value (like 
my_database.db
in the example). If you'd like to override the value of the setting at run time to use a different database file, you can set the 
TAP_SQLITE_DATABASE
environment variable when executing 
meltano elt
: http://meltano.com/docs/configuration.html#configuring-settings
t
ok. im getting an error: File "/Users/eliet/meltano-projects/my-meltano-project/.venv/lib/python3.8/site-packages/meltano/core/plugin/settings_service.py", line 110, in _meltano_yml_config   return { TypeError: 'str' object is not a mapping
d
Can you share your 
meltano.yml
?
t
sure
message has been deleted
d
YAML is pretty finicky about whitespace, and in this case the single space after 
database:
actually matters 🙂
It's currently interpreting 
database:my_database.db
as a string (hence the 
TypeError: 'str' object is not a mapping
error), instead of a 
key: value
mapping
t
ahhh ok
let me edit and try again
d
I recommend using an editor or IDE with syntax highlighting that will help you spot these kind of things
I can recommend VSCode
t
message has been deleted
d
OK, no more error! Do you see the 
my_database.db
file?
You can open it with any SQLite browser or the sqlite3 CLI
t
ok great. now i would like to know...when it reads- transformation skipped...why has it skipped? this is a csv moving to a database table...would THAT be the transformation
d
Transformation is a step that follows loading (in the case of ELT tools like Meltano -- in ETL tools it sits in between E and L): https://meltano.com/#transformation
Meltano uses dbt, which lets you write transformations in SQL, that acts on the original "raw" data in the loaded table and results in a new table
So you use transformations if the raw data schema coming out of the extractor is not to your liking, so that you can change the types of columns, rename them, join different tables, etc
t
thanks, that last post, where you explained transformation, was what i was looking to understand. thank you 🙂
d
Awesome 🙂
Open in Slack
PreviousNext
Powered by