Developing ElliottBrowser for Quandl.Com - Intro

▶ Brief description of ElliottBrowser

    ElliottBrowser is a tool for Elliott wave analysis, mainly based on the methods described in the Glenn Neely's book, 'Mastering Elliott Wave'.

    It's main functionality is to display 'structure-and-progress notations' of waves in chart data of stocks - See Fig 1.

    ElliottBrowser for Quandl.Com uses free data gathered from Quandl.Com, especially its WIKI database.

Fig 1. ElliottBrowser

▶ Usecases

    - Downloading / storing the list of the stocks in Quandl.Com's WIKI database
    - Refreshing the list when needed
    - Displaying the list
    - Downloading / storing chart data incrementally on-demand
    - Applying Elliott wave analysis to extract structure-and-progress notations of waves
    - Charting

▶ Prerequisites

   - .Net framework 4.5 or above
    - Microsoft SQL Server Compact 4.0
    - Visual Studio Community 2015
    - Quandl.Com account

Storing / Refreshing the List of the Stocks in WIKI database

    When a user using ElliottBrowser wants to view a chart of a stock, he first needs to lookup its ticker symbol (or shortly symbol) or to select an item in the list of stock symbols. So, ElliottBrowser provdes a panel for that purpose. See Fig 2.

Fig 2. The Panel of Symbols

    Quandl.Com provides the list of stock symbols in the WIKI database through the URL "https://www.quandl.com/api/v3/databases/WIKI/codes.json". When accessing the URL using a web browser, it saves the response data as a .zip file rather than displaying it as some form of JSON string.

    To prepare the panel of symbols, should ElliottBrowser download the .zip file and unzip to extract the symbols every time it is starting? It seems to be reasonable. But, what if the .zip file in Quandl.Com remain unchanged for a while?

▷ Functional Design

    1. When ElliottBrowser accesses Quandl.Com for the first time, it downloads and stores the .zip file containing the list of stock symbols to a local disk as it is. Of course, it should extract the contents of the .zip file and prepare the panel of symbols.

    2. After that, when it is starting, it initiates 2 threads, one for extracting symbols from the local .zip file to prepare the panel, and the other for fetching the remote .zip file to compare it with the local file. When they mismatch, the second thread saves the new .zip file and notifies so that ElliottBrowser can refresh the panel of symbols.

▷ Some Technical Specifics

    1. How to get the list of symbols

    The .zip file contains only one .csv file with each line corresponding to a symbol-description pair representing a stock as Fig 3 below. To get the list of symbols, we parse each line into a key-value pair, which will be added as an element in a dictionary data type.

Fig 3. Contents extracted from the .zip file

    private Dictionary _zipped_CSV_to_dic(ZipArchive za)
    {
        Dictionary dic = new Dictionary();
        using (var unzipped = za.Entries[0].Open())
        {
            using (var reader = new StreamReader(unzipped, Encoding.ASCII))
            {
                string line, symbol, name;
                string[] arr, arr1;
                dic = new Dictionary();

                while ((line = reader.ReadLine()) != null)
                {
                    arr = line.Trim().Split(',');
                    arr1 = arr[0].Split('/');
                    symbol = arr1.Length > 1 ? arr1[1] : arr[0];
                    name = line.Replace(arr[0] + ",", string.Empty).Replace("\"", string.Empty)
                                .Replace("Prices, Dividends, Splits and Trading Volume", string.Empty).Trim();
                    dic[symbol] = name;
                }
            }
        }
        return dic;
    }

    2. Downloading and comparing the .zip file with the local .zip previously stored

    The .zip file is fetched as a content in a HttpWebResponse, and is extracted to get the .csv file using ZipArchive class in System.IO.Compression library in .Net 4.5. After that, using MD5 hash in System.Security.Criptyography library, the .csv is compared with that contained in the previously stored .zip file.

    delegate void d_refreshSymbols(string symbols_fstr, string database, ref Dictionary dic, ref int err);

    private void _refresh_symbols(string symbols_fstr, string database, ref Dictionary dic, ref int err)
    {
        string url, path;
        HttpWebResponse response = default(HttpWebResponse);
        url = string.Format(symbols_fstr, database);

        var webRequest = (HttpWebRequest)WebRequest.Create(url);
        try
        {
            response = webRequest.GetResponse() as HttpWebResponse;

            if (response.ContentType == "application/zip")
            {
                bool to_serialize = true, overwrite = false;
                path = _home + Path.DirectorySeparatorChar + database;
                if (!Directory.Exists(path)) Directory.CreateDirectory(path);
                path += Path.DirectorySeparatorChar + response.ResponseUri.AbsolutePath.Substring(1);
                Properties.Settings.Default.symbolsPath = path; Properties.Settings.Default.Save();

                using (var mstr = new MemoryStream())
                using (var str = response.GetResponseStream())
                {
                    str.CopyTo(mstr);
                    ZipArchive zipArchive = new ZipArchive(mstr, ZipArchiveMode.Read);

                    if (File.Exists(path))
                    {
                        using (Stream input = File.OpenRead(path))
                        {
                            var serialized = new ZipArchive(input, ZipArchiveMode.Read);
                            using (var md5 = MD5.Create())
                            {
                                var h1 = System.Text.Encoding.UTF8.GetString(md5.ComputeHash(zipArchive.Entries[0].Open()));
                                var h2 = System.Text.Encoding.UTF8.GetString(md5.ComputeHash(serialized.Entries[0].Open()));
                                to_serialize = overwrite = (h1 != h2);
                            }
                        }
                    }

                    if (to_serialize)
                    {
                        mstr.Position = 0;
                        using (Stream output = File.OpenWrite(path))
                            mstr.CopyTo(output);

                        if (overwrite)
                        {
                            if (SymbolsUpdate != null) SymbolsUpdate(); err = 0; goto step1;
                        }
                    }

                    dic = _zipped_CSV_to_dic(zipArchive);
                }
    step1:
                err = 0;
            }
            else
            {
                err = (int)APIError.Unknown;
            }
        }
        catch (WebException ex)
        {
            QuandlError qerror = GetQuandlError(ex);
            err = qerror.ToInt();
        }
    }

 

    3. Getting symbols from local, refreshing with one from Quandl

    Once the .zip file being stored locally, ElliottBrowser prepares its panel of symbols using the local file first, while invokes a delegate to check the mismatch between the local and the remote .zip.

    public void RefreshSymbols(out Dictionary dic, out int err)
    {
        string ppath; // The path to the previously stored .zip file
        symbols_fstr = "https://www.quandl.com/api/v3/databases/{0}/codes.json";
        string database = "WIKI";
        dic = new Dictionary();
        err = 0;

        ppath = Properties.Settings.Default.symbolsPath;
        if (!string.IsNullOrEmpty(ppath))
        {
            if (File.Exists(ppath) && !File.GetAttributes(ppath).HasFlag(FileAttributes.Directory))
            {
                string[] parr = ppath.Split(Path.DirectorySeparatorChar);
                if (parr[parr.Length - 2] == database)
                {
                    using (Stream input = File.OpenRead(ppath))
                    {
                        ZipArchive zipArchive = new ZipArchive(input, ZipArchiveMode.Read);
                        dic = _zipped_CSV_to_dic(zipArchive);
                    }

                    d_refreshSymbols d = _refresh_symbols;
                    d.BeginInvoke(symbols_fstr, database, ref dic, ref err, null, null);
                    err = 0; return;
                }
            }
        }

        _refresh_symbols(symbols_fstr, database, ref dic, ref err);
    }

※  The sources of ElliottBrowser for Quandl.Com and its necessary libraries have been published in a GitHub repository.