csv2sqlite

Description

csv2sqlite converts CSV files into SQLite databases.

• Delimiters and enclosures can be any Unicode string: <#>, an emoji, a multi-byte invisible separator. They're detected automatically, along with the line ending.
• Fast, with low memory usage.
• Handles extremely large files.

sqlite-compare checks two SQLite files against each other and stops at the first difference.

Installation

Build from source, see Compilation.

Compilation

Requirement is a working Go installation, see https://go.dev

csv2sqlite.

make csv2sqlite
# or for mac
make csv2sqlite_mac

sqlite-compare.

make sqlite-compare

Usage csv2sqlite

Usage

Name / Shorthand	Default	Usage
--version	false	Prints the version
--input -i		Input CSV file
--output -o		Output CSV (or SQLite) file
--delimiter -d		Set the delimiter in case auto-detect should not be used. Can be one of: APOSTROPHE ASTERISK CARET COMMA DOLLAR EMPTY INFORMATION_SEPARATOR_TWO INVISIBLE_SEPARATOR PIPE POUND QUOTATION_MARK QUOTATION_MARK_DOUBLE QUOTATION_MARK_TRIPLE RECORD_SEPARATOR RECORD_SEPARATOR_SB SEMICOLON TAB TILDE UNIT_SEPARATOR
--enclosure -e		Set the enclosure in case auto-detect should not be used, can be one of: APOSTROPHE ASTERISK CARET COMMA DOLLAR EMPTY INVISIBLE_SEPARATOR PIPE POUND QUOTATION_MARK QUOTATION_MARK_DOUBLE QUOTATION_MARK_TRIPLE RECORD_SEPARATOR RECORD_SEPARATOR_SB RECORD_SEPARATOR_SB SEMICOLON TAB TILDE UNIT_SEPARATOR
--start	0	The starting position where the header is, useful for CSVs produced by Excel. Only works in combination with newline delimited, -nd enabled if set.
--skip	-1	Rows to skip after the header
--limit	-1	Amount of rows to process
--literal-delimiter		Sets the delimiter to use to a literal string, if set the --delimiter is ignored
--literal-enclosure		Sets the enclosure to use to a literal string, if set the --enclosure is ignored
--php	false	CSV file was produced by PHP, enclosure escaped by backslash
--exclude	[]	Column indices to exclude from conversion
--newline		Set the newline to be used in case auto-detection should not be used. Can be one of: LF CR CRLF
--sqlite	false	Output SQLite file
--nd	false	Input CSV is newline delimited and does not have newlines in the middle of a field
--break-count	15000	Causes the sqlite writer to stop writing at the specified amount of rows. Can cause the merge to be triggered earlier. The first writer stops at N/3
--commit-rate -r	5000	Flush rate for comitting rows to disk
--buffer-size -b	15625	Buffer size for reading from disk in kilobytes
--threads	16	Threads to use when writing output, default is set to the amount of cores
--channel-capacity	8000	Buffered channels capacity multiplied by threads
--chunk-size	10	Chunksize before pushing the data to the channel
--cpu-profile		Filename for a cpu-profile ouput for pprof
--heap-profile		Filename for a heap-profile ouput for pprof
--trace-profile		Filename for a trace profile for go tool trace
--fgprofile		Filename for a fgprofile ouput for pprof, see https://github.com/felixge/fgprof
--log-header	0	Logs the header for the collected data, pass 1 to output it. Can be used together with seq
--quiet -q	false	Quiet mode do not output performance data
--csv-report	false	Performance data output in TSV format, default is JSON
--fixed-eol	false	EOL character (CR/CRLF/LF) doesn’t change after auto-detect
--drop	false	Drops all outgoing data, for perf testing disk reading
--rowid	false	When the order of the CSV file must be kept the same in the resulting SQLite, this flag will cause each row in the SQLite to receive a RowID for SQLite.
--progress	false	Shows progress
--scratch-dir		Directory for storing the smaller SQLite files before merging them to the target. Same as output if left blank.
--debug	false	Outputs debug information, careful is extremely chatty.
--rw-debug	false	Outputs record writer debug information, careful is extremely chatty.
--log-csv-reader	false	Logs the occurence and data where the Go built-in CSV Reader is used. Can be used without the debug mode.
--log-dropped	false	Outputs dropped data
--overflow	false	Overflows the data. Will abort processing.

Usage sqlite-compare

Usage

Name / Shorthand	Default	Usage
--sqlite-a -a		SQLite file A to compare with
--sqlite-b -b		SQLite file B to compare against
--table -t	data	Table to compare
--null	false	If null is set to false it compares using TEXT fields and a NULL is automatically converted to an empty text field. If this is unwanted set null to be true.
--write-results	false	Writes a result file. $a.passed when no difference was found. $a.failed when a difference was found, file will contain the error.
--progress	false	Shows progress
--ignore-count	false	Ignore the rowcount, show the rows that are different
--max-errors	0	Maximum amount of errors to allow before exiting
--limit	50000	Row limit per query
--no-ignore-crlf	false	Do not ignore CRLF difference from A
--no-ignore-cr	false	Do not ignore CR difference from A
--no-ignore-enclosure	false	Do not strip enclosures from both to ignore
--no-escape-ignore	false	Do not ignore incorrect detected enclosures from A to B (PHP parsing error)
--no-sort	false	Do not order data for indexed columns
--no-column-intersect	false	Do not intersect on column names when they differ
--enclosure -e	"	Enclosure to use
--threads	16	Thread limit
--cpu-profile		Filename for a cpu-profile ouput for pprof
--heap-profile		Filename for a heap-profile ouput for pprof

Documentation

Currently no additional documentation is yet available.

Inside docs/ there is a file readme.adoc which can be converted to Markdown for Github using

README.MD.

make markdown

Configuration

The only configuration available are the parameters during runtime, see ???

Testing, Code Coverage, Static Analysis

Run the tests with coverage:

go test -cover -coverprofile coverage.out ./... -count=1

Deployment

See Installation

Versioning

This library follows semantic versioning.