The default result window is 10 and we need to explicitly specify (10000 here) for all the results to be dumped into the resonse obj.
If not you will get only partial results.

Printing just the unique subjects and their count.

Converting the resulting dictionary elements into a list and counting the no. of occurences of each value is
inefficient when you have a dictionary object to operate on already.

Dictionaries use hashtables, using key value-hash pairs they can make accessing a value efficient
but arrays are sequentially stored and randomly accessed, making the lookups slow when you have large no. of values.

purpose: To see unique subjects and their no. of occurences so that we can get a handle on the hot topic.

This piece checks and connects to elastic instance if it is up and does some
basic queries and returns a json formatted result. More useful queries coming up.

Merge remote-tracking branch 'louistw/master'

Modified date format to include timezone

See merge request louistw/lsoftarchive2json!7

Merge remote-tracking branch 'louistw/master'

Replacing non-breaking space to real space is enough to fix the problem
that stops the email parser

To extract reply content only, eliminate the previous messages

Since support-watch has more message in a month, 2 digits is not enough.

Since ElasticSearch is case-sensitive in field name

Using BeautifulSoup to extract content from text/html content type

Date format has to be changed to match the type accepted for mapping

Changed fields selected to only Date, From, Subject, and Body
Formatted Date field, which accepted by ElasticSearch
Formatted From field, which removed unnecessary double-quote

Conflicts:
	spliter.sh

Mac uses {999} and other systems use {*} in csplit utility.

Made the parser output only if the message is parsed correctly and
has content

Changed some syntax to be runable in python3

It will run with option {999} only on Mac, e.g. uanme -s is 'Darwin'

Script now can take more than one filename as arguments

Added permissions for direct execute of script

Master

See merge request louistw/lsoftarchive2json!5

Update spliter.sh

See merge request louistw/lsoftarchive2json!4

Removed the line responsible for adding the metadata to the Raw json. It is needed if we load json manually into the elastic search, but jsonpyes takes care of it implicitly.

The raw json we get from parser is just enough to be used with the jsonpyes tool as an input.

input unrestricted to .txt file.

Fixed script path and made executable

See merge request louistw/lsoftarchive2json!2

Added env support for finding python and permissions for direct
execute of script.

Conflicts:
	parser.py

Only continue working on extracting data when the parser parses successfully

Check both content type and filename for the body
Since only attached file has filename, email body does not