<blockquote>
<p>Here are some of the tools I use to process and clean data from all manner of customers:</p>
</blockquote>
<p><a href="https://linux.die.net/man/1/detox">detox</a></p>
<p>The <strong>detox</strong> utility renames files to make them easier to work with. It removes spaces and other such annoyances. It’ll also translate or cleanup Latin-1 (ISO 8859-1) characters encoded in 8-bit ASCII, Unicode characters encoded in UTF-8, and CGI escaped characters.</p>
<p>See other episodes for great sed information. I like to remove DOS end of line and end of file characters:</p>
<!-- <p><code>sed -i 's/ //g' *.txt</code> or <code>sed -i 's/\r//g' *.txt</code></p> -->
<pre><code>sed -i &#39;s/
//g&#39; *.txt</code></pre>
<p>or</p>
<pre><code>sed -i &#39;s/\r//g&#39; *.txt</code></pre>
<h2 id="command-line-tools">Command-line tools</h2>
<ul>
<li>ack</li>
<li>awk</li>
<li>detox</li>
<li>grep</li>
<li>pandoc</li>
<li>pdftotext -layout</li>
<li>sed</li>
<li>unix2dos and dos2unix</li>
<li>wget</li>
<li>curl</li>
</ul>
<h2 id="r-libraries">R libraries</h2>
<ul>
<li>RCurl</li>
<li>XML</li>
<li>rvest</li>
<li>tm</li>
<li>xlsx</li>
</ul>
<h2 id="python-libraries">Python libraries</h2>
<ul>
<li>beautifulsoup</li>
<li>csv</li>
<li>nltk <a href="https://www.youtube.com/playlist?list=PLQVvvaa0QuDf2JswnfiGkliBInZnIC4HL">YouTube Series</a></li>
<li>rdflib</li>
<li>re</li>
</ul>
<h2 id="vim-tricks">Vim tricks</h2>
<ul>
<li>buffer searches (<code>:vim /pattern/ ##</code>)</li>
<li>Ack plugin</li>
<li>bufdo (<code>:bufdo %s/pattern/replace/ge | update</code>)</li>
</ul>
<h2 id="other-tools">Other tools</h2>
<ul>
<li><a href="https://openrefine.org">OpenRefine</a></li>
<li><a href="https://okfnlabs.org/reconcile-csv/">reconcile-csv</a></li>
<li><a href="https://tabula.technology/">tabula</a></li>
</ul>
