Spiral journey over kubernetes, cloud and devops world. https://devopsspiral.com/ https://devopsspiral.com/articles/robotframework/rf-reports-azureblob/ Thu, 08 Apr 2021 00:00:00 +0000 <p>In a long way off goal of making <a href="https://github.com/devopsspiral/rf-service">rf-service</a> covering everything between writing tests and viewing test results I decided to work a little more on where the test reports could be published. Using managed service for that seems to be perfect choice, especially if you could handle it in a ‘send and forget’ manner. In my work we are using Azure extensively so Azure Blob storage with its static website hosting is what I decided to use. In this article you will find very basic introduction to static web hosting on Azure and how you can send Robot Framework reports there programmatically.</p> <div class="panel radius"> <p id="toc"><strong>Table of Contents</strong></p> <ul id="markdown-toc"> <li><a href="#context" id="markdown-toc-context">Context</a></li> <li><a href="#azure-blob-storage-static-website-hosting" id="markdown-toc-azure-blob-storage-static-website-hosting">Azure Blob Storage static website hosting</a> <ul> <li><a href="#creating-blob-storage-and-static-website" id="markdown-toc-creating-blob-storage-and-static-website">Creating blob storage and static website</a></li> </ul> </li> <li><a href="#changes-in-rf-service" id="markdown-toc-changes-in-rf-service">Changes in rf-service</a> <ul> <li><a href="#sending-reports-to-azure-using-python" id="markdown-toc-sending-reports-to-azure-using-python">Sending reports to Azure using Python</a></li> </ul> </li> <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> </ul> </div> <h2 id="context">Context</h2> <p>This article is a part of series connected with testing on Kubernetes. You can find more info in following articles:</p> <p><a href="https://devopsspiral.com/articles/k8s/robotframework-kubelibrary/">Robot Framework library for testing Kubernetes</a> - in this part I’m describing Robot Framework library (Python) that uses Kubernetes client for getting info about your cluster and turning it into actual test suites.</p> <p><a href="https://devopsspiral.com/articles/k8s/robotframework-service/">Testing on kubernetes - rf-service</a> - this article describes Python service executed in a form of CronJob that actually runs the tests from KubeLibrary on kubernetes cluster.</p> <p><a href="https://devopsspiral.com/articles/k8s/robotframework-service-fe/">Intro to Vue.js. Testing on kubernetes - rf-service frontend</a> - adding frontend to the rf-service.</p> <p><a href="https://devopsspiral.com/articles/k8s/testing-with-octopus/">Testing with octopus</a> - using Octopus project for test execution.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="azure-blob-storage-static-website-hosting">Azure Blob Storage static website hosting</h2> <p>Azure blob storage is a great way of keeping files in cloud. It’s a simplified abstraction of file system with API allowing to configure things like retention, access control, redundancy etc. It is part of more general concept - storage account - which can be used for services like queues, tables or even big data scenarios like Data Lake storage.</p> <h3 id="creating-blob-storage-and-static-website">Creating blob storage and static website</h3> <p>Creating blob storage is pretty easy, as a prerequisite, you need to have resource group in place, as a container for your storage account. Depending on the requirements you can select performance, account type and proper replication level. I will go with the defaults.</p> <p><img src="/images/rf-report-blob/rf-blob-sa.png" alt="Creating Storage Account V2" /></p> <p>After storage account is ready to be used, just go to Static website and enable the feature. This is exactly the thing that will make Robot Framework html reports to be visible as a website. You don’t need to set index or error document in this case, we won’t be really serving any startpage or anything like this, just the reports. It is good to take note of the primary endpoint, this is the URL under which your content will be served.</p> <p><img src="/images/rf-report-blob/static-website.png" alt="Enabling static website" /></p> <p>Enabling static website will create <em>$web</em> container, which is kind of root of everything you want to publish.</p> <p><img src="/images/rf-report-blob/containers.png" alt="Web container" /></p> <p>Now the important part, by default static websites are public - even though blob storage is by default not accessible publicly the $web container is handled differently (it supposed to be serving website ). You can limit the access to it in different ways, in corporate environment you would probably have private network connecting your work environment with Azure, network of internal/public IPs, etc. There is quick and easy way to limit the access only to your PC though, you can get back to storage account level, go to Networking and allow access from selected networks and use button to add client IP address. This will configure firewall to let in only your IP (remember that this IP may change depending on how exactly you are connecting to the internet).</p> <p><img src="/images/rf-report-blob/firewall.png" alt="Web container" /></p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="changes-in-rf-service">Changes in rf-service</h2> <p>Looking at the rf-service changes, simply new publisher called AzureBlobPublisher is available.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">usage: rf-service <span class="o">[</span><span class="nt">-h</span><span class="o">]</span> <span class="o">[</span><span class="nt">-i</span> INCLUDE_TAGS] <span class="o">[</span><span class="nt">-e</span> EXCLUDE_TAGS] <span class="o">[</span><span class="nt">--AzureBlobPublisher-connection_string</span> AZUREBLOBPUBLISHER_CONNECTION_STRING] <span class="o">[</span><span class="nt">--AzureBlobPublisher-path</span> AZUREBLOBPUBLISHER_PATH] <span class="o">[</span><span class="nt">--AzureBlobPublisher-prefix</span> AZUREBLOBPUBLISHER_PREFIX] <span class="o">[</span><span class="nt">--AzureBlobPublisher-blob_url</span> AZUREBLOBPUBLISHER_BLOB_URL] <span class="o">[</span><span class="nt">--CaddyPublisher-url</span> CADDYPUBLISHER_URL] <span class="o">[</span><span class="nt">--LocalPublisher-dest</span> LOCALPUBLISHER_DEST] <span class="o">[</span><span class="nt">--LocalFetcher-src</span> LOCALFETCHER_SRC] <span class="o">[</span><span class="nt">--ZipFetcher-url</span> ZIPFETCHER_URL] <span class="o">[</span><span class="nt">--ZipFetcher-path</span> ZIPFETCHER_PATH] <span class="o">[</span>config_file] RobotFramework service. positional arguments: config_file JSON config file optional arguments: <span class="nt">-h</span>, <span class="nt">--help</span> show this <span class="nb">help </span>message and <span class="nb">exit</span> <span class="nt">-i</span> INCLUDE_TAGS, <span class="nt">--include</span> INCLUDE_TAGS Include <span class="nb">test </span>tags <span class="nt">-e</span> EXCLUDE_TAGS, <span class="nt">--exclude</span> EXCLUDE_TAGS Exclude <span class="nb">test </span>tags <span class="nt">--AzureBlobPublisher-connection_string</span> AZUREBLOBPUBLISHER_CONNECTION_STRING <span class="nt">--AzureBlobPublisher-path</span> AZUREBLOBPUBLISHER_PATH <span class="nt">--AzureBlobPublisher-prefix</span> AZUREBLOBPUBLISHER_PREFIX <span class="nt">--AzureBlobPublisher-blob_url</span> AZUREBLOBPUBLISHER_BLOB_URL <span class="nt">--CaddyPublisher-url</span> CADDYPUBLISHER_URL <span class="nt">--LocalPublisher-dest</span> LOCALPUBLISHER_DEST <span class="nt">--LocalFetcher-src</span> LOCALFETCHER_SRC <span class="nt">--ZipFetcher-url</span> ZIPFETCHER_URL <span class="nt">--ZipFetcher-path</span> ZIPFETCHER_PATH</code></pre></figure> <p>It has 4 parameters described as:</p> <ul> <li><strong>connection_string</strong> - connection string taken from storage account Access keys (see below picture)</li> <li><strong>path</strong> - path in the $web container, can be used for grouping reports</li> <li><strong>prefix</strong> - prefix for the target report, can be used for test type distinction</li> <li><strong>blob_url</strong> - entirely informative parameter. Blob url along with path and prefix will write complete report url in logs</li> </ul> <p><img src="/images/rf-report-blob/connection_string.png" alt="Container string source" /></p> <p>Example test execution with publishing reports in Azure Blob Storage would look similar to:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">./src/scripts/rf-service <span class="nt">--AzureBlobPublisher-path</span> some/path/ <span class="nt">--AzureBlobPublisher-prefix</span> smoke <span class="nt">--AzureBlobPublisher-connection-string</span> <span class="s1">'DefaultEndpointsProtocol=...'</span> <span class="nt">--LocalFetcher-src</span> <span class="nb">test</span>/resources/testcases/</code></pre></figure> <p>And the report would be published under <a href="localhost">https://v2generalpurpose.z20.web.core.windows.net/some/path/smoke15-03-21T175232-662444.html</a></p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="sending-reports-to-azure-using-python">Sending reports to Azure using Python</h3> <p>It is worth to go through the Python code that is responsible for sending the reports in case you want to use it outside the rf-service.</p> <figure class="highlight"><pre><code class="language-python" data-lang="python"> <span class="k">def</span> <span class="nf">_publish</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">result</span><span class="p">):</span> <span class="n">reportname</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">get_standard_reportname</span><span class="p">()</span> <span class="n">upload_file_path</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'/tmp/</span><span class="si">{</span><span class="n">reportname</span><span class="si">}</span><span class="s">.html'</span> <span class="n">ResultWriter</span><span class="p">(</span><span class="n">result</span><span class="p">).</span><span class="n">write_results</span><span class="p">(</span><span class="n">report</span><span class="o">=</span><span class="n">upload_file_path</span><span class="p">,</span> <span class="n">log</span><span class="o">=</span><span class="bp">None</span><span class="p">)</span> <span class="n">blob_service_client</span> <span class="o">=</span> <span class="n">BlobServiceClient</span><span class="p">.</span><span class="n">from_connection_string</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">connection_string</span><span class="p">)</span> <span class="n">container_name</span> <span class="o">=</span> <span class="s">"$web"</span> <span class="n">target_blob</span> <span class="o">=</span> <span class="sa">f</span><span class="s">"</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">path</span> <span class="ow">or</span> <span class="s">''</span><span class="si">}{</span><span class="bp">self</span><span class="p">.</span><span class="n">prefix</span> <span class="ow">or</span> <span class="s">''</span><span class="si">}{</span><span class="n">reportname</span><span class="si">}</span><span class="s">.html"</span> <span class="n">blob_client</span> <span class="o">=</span> <span class="n">blob_service_client</span><span class="p">.</span><span class="n">get_blob_client</span><span class="p">(</span><span class="n">container</span><span class="o">=</span><span class="n">container_name</span><span class="p">,</span> <span class="n">blob</span><span class="o">=</span><span class="n">target_blob</span><span class="p">)</span> <span class="n">static_html_content_settings</span> <span class="o">=</span> <span class="n">ContentSettings</span><span class="p">(</span><span class="n">content_type</span><span class="o">=</span><span class="s">'text/html'</span><span class="p">)</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">upload_file_path</span><span class="p">,</span> <span class="s">"rb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">data</span><span class="p">:</span> <span class="n">blob_client</span><span class="p">.</span><span class="n">upload_blob</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">content_settings</span><span class="o">=</span><span class="n">static_html_content_settings</span><span class="p">)</span> <span class="bp">self</span><span class="p">.</span><span class="n">publish_target</span> <span class="o">=</span> <span class="sa">f</span><span class="s">'</span><span class="si">{</span><span class="bp">self</span><span class="p">.</span><span class="n">blob_url</span> <span class="ow">or</span> <span class="s">""</span><span class="si">}</span><span class="s">/</span><span class="si">{</span><span class="n">target_blob</span><span class="si">}</span><span class="s">'</span></code></pre></figure> <p>Going from top to bottom, first of all I’m writing Robot Framework report to file which then will be uploaded to Azure Blob storage. Then, there is part containing initialization of blob service client and setting target blob path and name. Remember to set content settings to ‘text/html’ to make sure that report is rendered as static page. Last step is just sending the content and viewing the url of the published report.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="conclusion">Conclusion</h2> <p>Azure blob storage is an interesting option for keeping Robot Framework reports. There are couple of functionalities like retention, tagging, access control that can be configured for your reports but were not described in this article, but even without it, simple publishing in managed service is very convenient. Microsoft makes sure that API for blob storage is accessible through different programming languages like Python, JavaScript, Ruby and others so it can become storage for different kind of automation.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> https://devopsspiral.com/articles/robotframework/rf-reports-azureblob/ https://devopsspiral.com/video/ansible-ala/ Tue, 24 Nov 2020 00:00:00 +0000 https://devopsspiral.com/video/ansible-ala/ https://devopsspiral.com/articles/linux/bcc-tools/ Wed, 21 Oct 2020 00:00:00 +0000 <p>After my first article about eBPF (<a href="https://devopsspiral.com/articles/linux/ebpf-unlock/">Unlocking eBPF power</a>), which was just-for-fun type of article, I wanted to start using eBPF in my daily work. I remember the situation from one of my interviews, where I was asked ‘How would I approach unknown binary/script?’. They were asking what tools I could use to get the most info about it and possibly perform some basic troubleshooting. I remember it quite vividly, mainly because I failed on it so badly. After reading this article you should probably be at much better position when dealing with similar type of problem.</p> <div class="panel radius"> <p id="toc"><strong>Table of Contents</strong></p> <ul id="markdown-toc"> <li><a href="#wait-what-is-ebpf-again" id="markdown-toc-wait-what-is-ebpf-again">Wait, what is eBPF again?</a></li> <li><a href="#do-i-need-to-be-badass-kernel-developer-to-use-ebpf" id="markdown-toc-do-i-need-to-be-badass-kernel-developer-to-use-ebpf">Do I need to be badass kernel developer to use eBPF?</a></li> <li><a href="#installation" id="markdown-toc-installation">Installation</a></li> <li><a href="#inspection" id="markdown-toc-inspection">Inspection</a> <ul> <li><a href="#execsnoop" id="markdown-toc-execsnoop">execsnoop</a></li> <li><a href="#tcpconnect" id="markdown-toc-tcpconnect">tcpconnect</a></li> <li><a href="#opensnoop" id="markdown-toc-opensnoop">opensnoop</a></li> <li><a href="#pythoncalls" id="markdown-toc-pythoncalls">pythoncalls</a></li> <li><a href="#trace" id="markdown-toc-trace">trace</a></li> </ul> </li> <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> </ul> </div> <h2 id="wait-what-is-ebpf-again">Wait, what is eBPF again?</h2> <p>If you hear eBPF for the first time I would recommend going through (<a href="https://devopsspiral.com/articles/linux/ebpf-unlock/">Unlocking eBPF power</a>), where I’m describing my first contact with eBPF. For quick recap: “eBPF is a functionality of linux kernel that allows lightweight execution of user code as a response to kernel events. The events could be hardware/software events, tracing events both static (compiled into code) and dynamic (attached in runtime), etc. The code itself is limited in a sense that it is guaranteed to finish (no loops) and is verified before loading into kernel.”</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="do-i-need-to-be-badass-kernel-developer-to-use-ebpf">Do I need to be badass kernel developer to use eBPF?</h2> <p>NO! And this it what I’ll try to show in this article. Reading some comments regarding eBPF I see people falling into two groups: the ones who doesn’t know eBPF yet and those using it for some serious stuff and describing it with complicated language. This can be very intimidating. Concept of building a bridge between user and kernel space is very powerful and there are many complex projects around eBPF. C language is also very powerfully, but how often do you think about grep or curl internals? Guess what, eBPF has also project called <a href="https://github.com/iovisor/bcc">BCC</a>(BPF Compiler Collection) that has around 100 tools. To be honest BCC is made for writing eBPF programs and the tools part is “just” addition.</p> <p>I will be using BCC toolkit for inspecting ansible, which is what I’m focused on at my work currently, but everything described here could be used for any other python based tools and most of it for practically any executable.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="installation">Installation</h2> <p>Official packages for Ubuntu seem to be outdated so I followed installation from source as described in <a href="https://github.com/iovisor/bcc/blob/master/INSTALL.md#ubuntu---source">here</a></p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nt">-y</span> <span class="nb">install </span>bison build-essential cmake flex git libedit-dev <span class="se">\</span> libllvm6.0 llvm-6.0-dev libclang-6.0-dev python zlib1g-dev libelf-dev git clone https://github.com/iovisor/bcc.git <span class="nb">mkdir </span>bcc/build<span class="p">;</span> <span class="nb">cd </span>bcc/build cmake .. make <span class="nb">sudo </span>make <span class="nb">install </span>cmake <span class="nt">-DPYTHON_CMD</span><span class="o">=</span>python3 .. <span class="c"># build python3 binding</span> <span class="nb">pushd </span>src/python/ make <span class="nb">sudo </span>make <span class="nb">install popd</span></code></pre></figure> <p>Since BCC has Python front-end, most of the tools are essentially Python scripts and needs valid PYTHONPATH env variable setting. You can export it once (in. bashrc/.zshrc) to avoid missing modules problems. Assuming that you’ve installed it in your home, you should be fine with below export.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">PYTHONPATH</span><span class="o">=</span><span class="nv">$PYTHONPATH</span>:~/bcc/src/python/</code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="inspection">Inspection</h2> <h3 id="execsnoop">execsnoop</h3> <p>To be perfectly honest ansible execution in my case is wrapped in a script that does some extra steps. I don’t know what steps exactly, but this can be checked using execsnoop which traces all new processes.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./the_ansible_wrapper.sh &amp; <span class="nb">sudo</span> ~/bcc/tools/execsnoop.py PCOMM PID PPID RET ARGS git 2345 2333 0 /usr/bin/git checkout develop <span class="o">(</span>…<span class="o">)</span> ansible-galaxy 2333 2310 0 ansible-galaxy <span class="nb">install</span> <span class="nt">-r</span> ansible/playbooks/requirements.yml <span class="nt">-f</span> <span class="nt">-v</span> <span class="o">(</span>…<span class="o">)</span> ansible-playboo 2344 2331 0 ansible-playbook <span class="nt">-i</span> inventory.py some_playbook.yml <span class="o">(</span>…<span class="o">)</span> python3.9 2356 2351 0 /usr/local/bin/python3.9 inventory.py <span class="nt">--list</span> <span class="o">(</span>…<span class="o">)</span></code></pre></figure> <p>You can easily get some info about what is happening, there is repository checkout, ansible roles download, playbook execution and it is using dynamic inventory in form of script. I’ve trimmed the output on purpose to hide some of the exact values, but with full output you would get repo names, executable versions, full paths to files and all other good stuff.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="tcpconnect">tcpconnect</h3> <p>Don’t you have this uncomfortable thought for a second that whenever you install something new on your computer there is a chance that someone is gathering some info about you and sending it out?</p> <p>Well, you can have similar feeling whenever you are working on a code that is not yet known to you. Ansible is all about handling configuration of remote systems, but target inventory is not the only system it connects to. Playbooks can first gather input from configuration store, secret store, infrastructure, etc. You can track all of those using tcpconnect. I changed the IPs in the output to some meaningful names.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span><span class="nb">sudo</span> ~/bcc/tools/tcpconnect.py &amp; <span class="nv">$ </span>./the_ansible_wrapper.sh <span class="o">&gt;</span> /dev/null ID COMM FD ERR PATH PID COMM IP SADDR DADDR DPORT 6560 the_ansible_wrapper.sh 4 my_machine vault 443 6569 ssh 4 my_machine github.com 22 6586 ssh 4 my_machine github.com 22 <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="opensnoop">opensnoop</h3> <p>So we know what is going on under the hood of the tool, where it is connecting to, it would be great to know which files are being used. This is exactly what opensnoop is doing. Not sure which ansible.cfg is effectively used? - use opensnoop. Looking for some variable source files? - use opensnoop. Or maybe you are just refactoring your ansible code and trying to get rid of some unused roles and plays - by seeing which files are read you can easily see what is left.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./the_ansible_wrapper.sh &amp; <span class="nb">sudo</span> ~/bcc/tools/opensnoop.py <span class="nt">-n</span> ansible-pl ID COMM FD ERR PATH 15638 ansible-playboo 3 0 /tmp/ansible_pdrczx2p/vault.json <span class="o">(</span>...<span class="o">)</span> 15638 ansible-playboo 6 0 /tmp/ansible_pdrczx2p/ansible/group_vars/all.yaml <span class="o">(</span>...<span class="o">)</span> 15638 ansible-playboo 4 0 /home/ansible/.ansible.cfg <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p>It is worth noting that you might observe limitations of some of those tools that manifests itself with entries like “Possibly lost X samples” this is sign that there are too many data coming into the script from BPF and user space script is not keeping up. Usually it happens when you started tracing too many calls, try limiting the tool to some specific PID or process name. If it is still too much the only option is to filter items on BPF side. The scripts has BPF snippets defined as strings so you can try editing it after some trial and error. I did that for opensnoop to limit the files to specific directory that contained all the playbooks. See the commented lines below.</p> <figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">int</span> <span class="nf">trace_return</span><span class="p">(</span><span class="k">struct</span> <span class="n">pt_regs</span> <span class="o">*</span><span class="n">ctx</span><span class="p">)</span> <span class="p">{</span> <span class="n">u64</span> <span class="n">id</span> <span class="o">=</span> <span class="n">bpf_get_current_pid_tgid</span><span class="p">();</span> <span class="k">struct</span> <span class="n">val_t</span> <span class="o">*</span><span class="n">valp</span><span class="p">;</span> <span class="k">struct</span> <span class="n">data_t</span> <span class="n">data</span> <span class="o">=</span> <span class="p">{};</span> <span class="c1">// defining target directory</span> <span class="c1">// const char searchString[] = "/tmp/ansible_";</span> <span class="n">u64</span> <span class="n">tsp</span> <span class="o">=</span> <span class="n">bpf_ktime_get_ns</span><span class="p">();</span> <span class="n">valp</span> <span class="o">=</span> <span class="n">infotmp</span><span class="p">.</span><span class="n">lookup</span><span class="p">(</span><span class="o">&amp;</span><span class="n">id</span><span class="p">);</span> <span class="k">if</span> <span class="p">(</span><span class="n">valp</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// missed entry</span> <span class="k">return</span> <span class="mi">0</span><span class="p">;</span> <span class="p">}</span> <span class="n">bpf_probe_read_kernel</span><span class="p">(</span><span class="o">&amp;</span><span class="n">data</span><span class="p">.</span><span class="n">comm</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">comm</span><span class="p">),</span> <span class="n">valp</span><span class="o">-&gt;</span><span class="n">comm</span><span class="p">);</span> <span class="n">bpf_probe_read_user</span><span class="p">(</span><span class="o">&amp;</span><span class="n">data</span><span class="p">.</span><span class="n">fname</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">data</span><span class="p">.</span><span class="n">fname</span><span class="p">),</span> <span class="p">(</span><span class="kt">void</span> <span class="o">*</span><span class="p">)</span><span class="n">valp</span><span class="o">-&gt;</span><span class="n">fname</span><span class="p">);</span> <span class="c1">// comparing two strings if they don't match leave the function</span> <span class="c1">// if (memcmp(&amp;data.fname, searchString,8) != 0) {</span> <span class="c1">// return 0;</span> <span class="c1">// }</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="pythoncalls">pythoncalls</h3> <p>Sometimes you want to get a little more from the underlying code. BCC tools has the ucalls utility that can track static tracing markers for different high-level languages including python (pythoncalls is a wrapper for ucalls). It allows tracking number of calls to the function and its latency. The problem is that you need to have python build from source with <code class="language-plaintext highlighter-rouge">--with-dtrace</code> flag to add USDT (User Statically Defined Trace) probes. Bear with me, it is not that complicated.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>apt-get <span class="nb">install </span>systemtap-sdt-dev wget https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tgz <span class="nb">tar</span> <span class="nt">-xf</span> Python-3.9.0.tgz <span class="nb">cd </span>Python-3.9.0/ ./configure ––enable–optimizations <span class="nt">--with-dtrace</span> <span class="c"># we want to install it locally next to existing Python</span> <span class="nb">sudo </span>make altinstall</code></pre></figure> <p>You can verify if everything is fine by using another BCC tool called tplist, which displays kernel tracepoints and USDT probes that we just enabled. Ucalls is using function__entry and function__return probes.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>~/bcc/tools/tplist.py <span class="nt">-l</span> python3.9 /usr/local/bin/python3.9 python:line /usr/local/bin/python3.9 python:function__entry /usr/local/bin/python3.9 python:function__return /usr/local/bin/python3.9 python:import__find__load__start /usr/local/bin/python3.9 python:import__find__load__done /usr/local/bin/python3.9 python:audit /usr/local/bin/python3.9 python:gc__start /usr/local/bin/python3.9 python:gc__done</code></pre></figure> <p>Now, when you start your script again you should see nice statistics of used functions. There are different flags that you can use with ucalls, in the following example I just used simple call count and latency.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./the_ansible_wrapper.sh &amp; <span class="nb">sudo</span> ~/bcc/tools/pythoncalls.sh <span class="nt">-Lm</span> <span class="nv">$!</span> METHOD <span class="c"># CALLS TIME (ms)</span> <span class="o">(</span>...<span class="o">)</span> /usr/local/lib/python3.9/shutil.py.rmtree 1 65.35 /usr/local/lib/python3.9/logging/__init__.py.makeRecord 1089 66.35 /usr/local/lib/python3.9/ssl.py.read 1 79.20 /usr/local/lib/python3.9/ssl.py.recv_into 1 79.22 /usr/local/lib/python3.9/socket.py.readinto 1 79.24 /usr/local/lib/python3.9/http/client.py._read_status 1 79.27 /usr/local/lib/python3.9/http/client.py.begin 1 79.85 /usr/local/lib/python3.9/http/client.py.getresponse 1 79.94 <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="trace">trace</h3> <p>Last thing that is missing from python script inspection perspective are the environment variables. The thing is a bit tricky, because there are possibly many ways programs are getting environment variables from the system. For C based tools it is usually getenv function from libc. So you could trace it using <code class="language-plaintext highlighter-rouge">ltrace -e getenv &lt;script&gt;</code>, or I should rather say <code class="language-plaintext highlighter-rouge">~/bcc/tools/trace.py 'r:c:getenv "%s=%s", arg1, retval'</code> to trace return probes (r:) of the libc library (c:) for function getenv. We want to print first argument and return value of the function which corresponds to env var name and value - it might need some tweaking as the output is not fitting into <em>trace</em> output columns but you get the idea. Python is handling those differently and it loads the env variables into dictionary kept in special _Environ object.</p> <p>If we are already building Python with USDT probes why not add another one dynamically to os.py module. I will use <a href="https://github.com/sthima/libstapsdt">libstapsdt</a> and python wrapper for it, called <a href="https://github.com/sthima/python-stapsdt">python-stapsdt</a>. This library is dynamically creating shared library which is exposing USDT probes so that tools like <em>trace</em> can see markers to which it can attach to. The code that needs to be added to os.py (/usr/local/lib/pyhon3.9/os.py in my case) can be found below.</p> <figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">importlib.util</span> <span class="n">spec</span> <span class="o">=</span> <span class="n">importlib</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">spec_from_file_location</span><span class="p">(</span><span class="s">"stapsdt"</span><span class="p">,</span> <span class="s">"/home/ansible/.local/lib/python3.9/site-packages/stapsdt.py"</span><span class="p">)</span> <span class="n">stapsdt</span> <span class="o">=</span> <span class="n">importlib</span><span class="p">.</span><span class="n">util</span><span class="p">.</span><span class="n">module_from_spec</span><span class="p">(</span><span class="n">spec</span><span class="p">)</span> <span class="n">spec</span><span class="p">.</span><span class="n">loader</span><span class="p">.</span><span class="n">exec_module</span><span class="p">(</span><span class="n">stapsdt</span><span class="p">)</span> <span class="n">provider</span> <span class="o">=</span> <span class="n">stapsdt</span><span class="p">.</span><span class="n">Provider</span><span class="p">(</span><span class="s">"pythonapp"</span><span class="p">)</span> <span class="n">probe</span> <span class="o">=</span> <span class="n">provider</span><span class="p">.</span><span class="n">add_probe</span><span class="p">(</span><span class="s">"getenvProbe"</span><span class="p">,</span> <span class="n">stapsdt</span><span class="p">.</span><span class="n">ArgTypes</span><span class="p">.</span><span class="n">uint64</span><span class="p">,</span> <span class="n">stapsdt</span><span class="p">.</span><span class="n">ArgTypes</span><span class="p">.</span><span class="n">uint64</span><span class="p">)</span> <span class="n">provider</span><span class="p">.</span><span class="n">load</span><span class="p">()</span> <span class="k">class</span> <span class="nc">_Environ</span><span class="p">(</span><span class="n">MutableMapping</span><span class="p">):</span> <span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">,</span> <span class="n">encodekey</span><span class="p">,</span> <span class="n">decodekey</span><span class="p">,</span> <span class="n">encodevalue</span><span class="p">,</span> <span class="n">decodevalue</span><span class="p">):</span> <span class="bp">self</span><span class="p">.</span><span class="n">encodekey</span> <span class="o">=</span> <span class="n">encodekey</span> <span class="bp">self</span><span class="p">.</span><span class="n">decodekey</span> <span class="o">=</span> <span class="n">decodekey</span> <span class="bp">self</span><span class="p">.</span><span class="n">encodevalue</span> <span class="o">=</span> <span class="n">encodevalue</span> <span class="bp">self</span><span class="p">.</span><span class="n">decodevalue</span> <span class="o">=</span> <span class="n">decodevalue</span> <span class="bp">self</span><span class="p">.</span><span class="n">_data</span> <span class="o">=</span> <span class="n">data</span> <span class="k">def</span> <span class="nf">getenv_probe</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="k">if</span><span class="p">(</span><span class="bp">self</span><span class="p">.</span><span class="n">encodekey</span><span class="p">(</span><span class="s">"PYTHON_INSPECT"</span><span class="p">)</span> <span class="ow">in</span> <span class="bp">self</span><span class="p">.</span><span class="n">_data</span><span class="p">):</span> <span class="k">while</span><span class="p">(</span><span class="ow">not</span> <span class="n">probe</span><span class="p">.</span><span class="n">is_enabled</span><span class="p">):</span> <span class="kn">import</span> <span class="nn">time</span> <span class="n">time</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="k">try</span><span class="p">:</span> <span class="n">value</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">_data</span><span class="p">[</span><span class="bp">self</span><span class="p">.</span><span class="n">encodekey</span><span class="p">(</span><span class="n">key</span><span class="p">)]</span> <span class="n">probe</span><span class="p">.</span><span class="n">fire</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="bp">self</span><span class="p">.</span><span class="n">decodevalue</span><span class="p">(</span><span class="n">value</span><span class="p">))</span> <span class="k">except</span> <span class="nb">KeyError</span><span class="p">:</span> <span class="n">probe</span><span class="p">.</span><span class="n">fire</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="s">""</span><span class="p">)</span> <span class="k">def</span> <span class="nf">__getitem__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="bp">self</span><span class="p">.</span><span class="n">getenv_probe</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> </code></pre></figure> <p>I’m importing stapsdt module, creating getenvProbe and running firing function as first thing in env variable dictionary getter. So whenever python code is trying to read env variable my probe will be fired. The extra <code class="language-plaintext highlighter-rouge">while(not probe.is_enabled):</code> loop allows waiting for the tracer to attach to the probe whenever PYTHON_INSPECT env variable is set. This is needed because environment variables are usually read as first thing and often BCC trace tool is not quick enough and ends up attaching after all the variables were already checked. Now you can see all the possible env variables and their values if set.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./the_ansible_wrapper.sh &amp; <span class="nb">sudo</span> ~/bcc/tools/trace.py <span class="nt">-p</span> <span class="nv">$!</span> <span class="s1">'u::getenvProbe "%s=%s", arg1, arg2'</span> <span class="o">(</span>...<span class="o">)</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">ANSIBLE_SKIP_TAGS</span><span class="o">=</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">ANSIBLE_USE_PERSISTENT_CONNECTIONS</span><span class="o">=</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">ANSIBLE_PRECEDENCE</span><span class="o">=</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">ANSIBLE_YAML_FILENAME_EXT</span><span class="o">=</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">ANSIBLE_NETCONF_SSH_CONFIG</span><span class="o">=</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">ANSIBLE_STRING_CONVERSION_ACTION</span><span class="o">=</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">ANSIBLE_VERBOSE_TO_STDERR</span><span class="o">=</span> 6686 6686 ansible-playboo getenvProbe <span class="nv">LANG</span><span class="o">=</span>C.UTF-8 6686 6686 ansible-playboo getenvProbe <span class="nv">PATH</span><span class="o">=</span>/home/ansible/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="conclusion">Conclusion</h2> <p>Hope that I was able to show how it is easy to use eBPF in your daily work. Obviously there are a lot more tools out there and writing your own script using BCC is probably the next step. Adding USDT probes in to the code opens the door to many possibilities regarding observability, audit and more - definitely need to think of little project regarding that.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> https://devopsspiral.com/articles/linux/bcc-tools/ https://devopsspiral.com/video/circleci-python/ Mon, 05 Oct 2020 00:00:00 +0000 https://devopsspiral.com/video/circleci-python/ https://devopsspiral.com/articles/linux/k3d-orb/ Wed, 23 Sep 2020 00:00:00 +0000 <p>I was trying to setup CI (CircleCI) for one of my repositories (<a href="https://github.com/devopsspiral/KubeLibrary">KubeLibrary</a>) and I needed k8s cluster to run test examples. I immediately wanted to setup k3d/k3s cluster, but it turned out not that easy as I thought. I ended up creating k3d orb which allows creating temporary k8s cluster just for tests. In this article I’m describing some of the internals of this orb implemented to fit into CircleCI. If you just want to use it follow orb <a href="https://circleci.com/orbs/registry/orb/devopsspiral/k3d">documentation</a>.</p> <div class="panel radius"> <p id="toc"><strong>Table of Contents</strong></p> <ul id="markdown-toc"> <li><a href="#background" id="markdown-toc-background">Background</a></li> <li><a href="#circleci-docker-setup" id="markdown-toc-circleci-docker-setup">CircleCI docker setup</a></li> <li><a href="#orb-design" id="markdown-toc-orb-design">Orb design</a> <ul> <li><a href="#helpers" id="markdown-toc-helpers">Helpers</a></li> <li><a href="#kubeconfigs" id="markdown-toc-kubeconfigs">Kubeconfigs</a></li> <li><a href="#k3d-cluster" id="markdown-toc-k3d-cluster">K3d cluster</a></li> <li><a href="#versions" id="markdown-toc-versions">Versions</a></li> <li><a href="#reaching-kubernetes-api" id="markdown-toc-reaching-kubernetes-api">Reaching kubernetes API</a></li> <li><a href="#running-your-own-containers-in-the-cluster" id="markdown-toc-running-your-own-containers-in-the-cluster">Running your own containers in the cluster</a></li> <li><a href="#exposing-services" id="markdown-toc-exposing-services">Exposing services</a> <ul> <li><a href="#expose-via-kubectl-proxy" id="markdown-toc-expose-via-kubectl-proxy">Expose via kubectl proxy</a></li> <li><a href="#expose-via-load-balancer" id="markdown-toc-expose-via-load-balancer">Expose via load balancer</a></li> <li><a href="#expose-via-ingress" id="markdown-toc-expose-via-ingress">Expose via ingress</a></li> </ul> </li> </ul> </li> <li><a href="#publishing-your-own-orb" id="markdown-toc-publishing-your-own-orb">Publishing your own orb</a></li> </ul> </div> <h2 id="background">Background</h2> <p>I was already writing about <a href="https://github.com/rancher/k3d">K3d</a> in <a href="https://devopsspiral.com/articles/k8s/k3d-skaffold/">this</a> article and I’m using it on daily basis for quick tests or development. Shortly speaking K3d is dockerized <a href="https://github.com/rancher/k3s">K3s</a> which in turn is a small (but certified) k8s distribution made by Rancher Labs.</p> <p>Starting a cluster is just creating containers with k8s nodes, so it shouldn’t be that difficult to make it work in docker environment supported by CircleCI. I’ve quickly learned that fitting k3d into CircleCI needs some gymnastics so I decided to create CircleCI orb to possibly make others people lives easier.</p> <p>CircleCI orb is just a way of packaging custom jobs, commands and executors. It doesn’t differ much from regular CI definition in .circleci/config.yml but it is allowing to hide some of the complexity.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="circleci-docker-setup">CircleCI docker setup</h2> <p>CircleCI is using remote docker environment, initialized by execution of <em>setup_remote_docker</em> command. Whatever docker command is used it will be executed on a separate vm, where the only interface is docker command itself. As a result, it has couple of implications regarding networking and file transfer - simply speaking, if you want to run anything towards docker containers it should be also running in a container.</p> <p>On top of that, separate steps executions are not sharing environment variables so there is no direct way of passing values outside the CircleCI command. One of the options is to simply write and read values between the commands.</p> <p>It is also worth mentioning that remote docker vm has its own resource limits. According to <a href="https://circleci.com/docs/2.0/building-docker-images/#specifications">documentation</a> it is 2xIntel(R) Xeon(R) @ 2.3GHz CPU, 8GB RAM and 100GB disk space. It is not too much, but should be enough to cover some of the scenarios and fits into K3s small deployment size (see <a href="https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/#cpu-and-memory">docs</a>).</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="orb-design">Orb design</h2> <h3 id="helpers">Helpers</h3> <p>To address some of the mentioned limitation and keep k8s experience similar to what you have on your local system I had to create helpers for kubectl and helm. They are all executed as docker containers running in k3d load balancer network to allow reaching kubernetes API on 0.0.0.0 for simplicity.</p> <p>Additionally to achieve persistency between steps I used couple of volumes for holding helm cache, config and data. Each kubectl and helm execution has also repository content mounted at /repo/ to allow easy application of YAML files or helm charts.</p> <p>All the setup is done in <code class="language-plaintext highlighter-rouge">k3d/k3d-helpers</code> which should be executed before first k3d cluster is created. This the place where the volumes are initialized.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker create <span class="nt">-v</span> /.kube <span class="nt">--name</span> kubeconfig alpine:3.4 /bin/true docker create <span class="nt">-v</span> /repo <span class="nt">--name</span> repo alpine:3.4 /bin/true docker <span class="nb">cp</span> <span class="nb">.</span> repo:/repo/ docker volume create helm_cache docker volume create helm_config docker volume create helm_data <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p>To hide kubectl and helm internals they both have wrappers implemented as bash functions kept in helpers.sh, which is then sourced in every <code class="language-plaintext highlighter-rouge">k3d/k3d-run</code> command.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">(</span>...<span class="o">)</span> <span class="nb">cat</span> <span class="se">\&lt;</span>&lt;EOF <span class="o">&gt;</span> helpers.sh <span class="nb">export </span><span class="nv">K3D_USING_HELPERS</span><span class="o">=</span><span class="nb">true export </span><span class="nv">K3D_CLUSTER</span><span class="o">=</span>NotSet <span class="nb">export </span><span class="nv">K3D_KUBECONFIG</span><span class="o">=</span>NotSet kubectl <span class="o">()</span> <span class="o">{</span> docker run <span class="nt">--rm</span> <span class="nt">--name</span> kubectl <span class="nt">-e</span> <span class="nv">KUBECONFIG</span><span class="o">=</span><span class="se">\$</span>K3D_KUBECONFIG <span class="se">\</span> <span class="nt">--network</span> container:k3d-<span class="se">\$</span><span class="o">{</span>K3D_CLUSTER<span class="o">}</span><span class="nt">-serverlb</span> <span class="se">\</span> <span class="nt">--volumes-from</span> kubeconfig <span class="se">\</span> <span class="nt">--volumes-from</span> repo <span class="se">\</span> bitnami/kubectl:<span class="se">\$</span><span class="o">{</span>KUBECTL_VERSION:-latest<span class="o">}</span> <span class="se">\$</span>@ <span class="o">}</span> <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="kubeconfigs">Kubeconfigs</h3> <p>Kubeconfigs are treated very similar to other data, each new cluster writes a file into /.kube/&lt;name of cluster&gt; on separate volume, which is also mounted into kubectl and helm containers. Path to kubeconfig is passed as <em>KUBECONFIG</em> environment variable. Switching of target cluster is done by <code class="language-plaintext highlighter-rouge">k3d/k3d-use</code> command that changes the value of <em>K3D_KUBECONFIG</em> globally. Creating new cluster always sets the KUBECONFIG to itself.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="k3d-cluster">K3d cluster</h3> <p>New k8s cluster is created using <code class="language-plaintext highlighter-rouge">k3d/k3d-up</code> command that accepts following parameters:</p> <ul> <li><em>cluster-name</em> - name of the cluster, used for determining kubeconfig, but also to point to correct cluster load balancer.</li> <li><em>k3s-version</em> - tag of the k3s container, used to select kubernetes version i.e.: v1.16.14-k3s1</li> <li><em>agents</em> - number of kubernetes agents, default: 1.</li> </ul> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="versions">Versions</h3> <p>Apart from kubernetes cluster API version, both kubectl and helm can be used in specific versions. It is configured by setting environment variables: <em>KUBECTL_VERSION</em> and <em>HELM_VERSION</em>. They can be set for particular step or globally for the job:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">(</span>...<span class="o">)</span> <span class="nb">jobs</span>: test-k3d-versions: executor: k3d/default environment: - KUBECTL_VERSION: 1.16.15 - HELM_VERSION: 3.3.1 steps: <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="reaching-kubernetes-api">Reaching kubernetes API</h3> <p>To reach kubernetes API directly it is best to place the container in the <em>k3d-${K3D_CLUSTER}-serverlb</em> network, this way the API is exposed on 0.0.0.0:6443. It is exactly how kubectl and helm containers are used.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">(</span>...<span class="o">)</span> helm <span class="o">()</span> <span class="o">{</span> docker run <span class="nt">--rm</span> <span class="nt">--name</span> helm <span class="nt">-e</span> <span class="nv">KUBECONFIG</span><span class="o">=</span><span class="se">\$</span>K3D_KUBECONFIG <span class="se">\</span> <span class="nt">--network</span> container:k3d-<span class="k">${</span><span class="nv">K3D_CLUSTER</span><span class="k">}</span><span class="nt">-serverlb</span> <span class="se">\</span> <span class="o">(</span>...<span class="o">)</span></code></pre></figure> <p>When k3d creates cluster it assigns random port that is forwarding traffic to 6443 and that is what is landing in kubeconfig eventually. Consequently, small change to cluster.server URL needs to be applied for each of the kubeconfig files to point to original 0.0.0.0:6443.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="running-your-own-containers-in-the-cluster">Running your own containers in the cluster</h3> <p>As I already said, anything that is supposed to target k3d cluster should run in the container, but what if the container is supposed to run in the cluster itself? Own container registry would be perfect, but it is not yet part of the orb. Fortunately, there is simpler approach introduced by k3d called image import. You can simply copy image that you just build into the cluster container cache like in the example below.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker build <span class="nt">-t</span> pyserver <span class="nt">-f</span> sample/own_image/Dockerfile <span class="nb">.</span> k3d image import <span class="nt">-c</span> <span class="k">${</span><span class="nv">K3D_CLUSTER</span><span class="k">}</span> pyserver:latest</code></pre></figure> <p>It is important to remember that to use container cache <em>imagePullPolicy</em> in kubernetes resources cannot be set to <em>Always</em> which enforces checking newer image in the registry. Example deployment would look similar to following code.</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"> <span class="na">template</span><span class="pi">:</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">app</span><span class="pi">:</span> <span class="s">pyserver</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">containers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">pyserver</span> <span class="na">imagePullPolicy</span><span class="pi">:</span> <span class="s">Never</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="exposing-services">Exposing services</h3> <p>After deploying your own workloads on the cluster it would be good to also know how to expose them to the outside world. In current k3d orb setup you can use at least 3 ways to achieve this. In all the examples I’m using Grafana chart as a target service and reaching it from busybox wget.</p> <h4 id="expose-via-kubectl-proxy">Expose via kubectl proxy</h4> <p>As a first option I will describe kubectl proxy which should handle all the networking complexity and expose the pod port on 0.0.0.0. In the k3d orb it is achieved by first getting pod name with kubectl and then using <em>proxy</em> command which is just kubectl in detached container. To reach target port you just need to deploy container in proxy container network as shown below (<code class="language-plaintext highlighter-rouge">--network container:proxy</code>).</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">- k3d/k3d-run: step-name: Expose service via kubectl proxy <span class="nb">command</span>: | helm repo add stable https://kubernetes-charts.storage.googleapis.com helm <span class="nb">install </span>grafana stable/grafana <span class="nb">sleep </span>30 <span class="nb">export </span><span class="nv">POD_NAME</span><span class="o">=</span><span class="si">$(</span>kubectl get pods <span class="nt">--namespace</span> default <span class="nt">-l</span> <span class="s2">"app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana"</span> <span class="nt">-o</span> <span class="nv">jsonpath</span><span class="o">=</span><span class="s2">"{.items[0].metadata.name}"</span><span class="si">)</span> proxy <span class="nt">--namespace</span> default port-forward <span class="nv">$POD_NAME</span> 3000 docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">--network</span> container:proxy busybox /bin/sh <span class="nt">-c</span> <span class="s2">"wget 0.0.0.0:3000; cat index.html"</span> | <span class="nb">grep </span>Grafana</code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h4 id="expose-via-load-balancer">Expose via load balancer</h4> <p>Another option is to use kubernetes service type LoadBalancer that will forward port on k3d-${K3D_CLUSTER}-serverlb container and forward it to each agent/node and to target pod eventually. SERVICE_IP is set to load balancer container and to be able to reach it we need to put our container (the one running the wget) in the k3d cluster network using <code class="language-plaintext highlighter-rouge">--network k3d-${K3D_CLUSTER}</code>.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">- k3d/k3d-run: step-name: Expose service via loadbalancer <span class="nb">command</span>: | helm repo add stable https://kubernetes-charts.storage.googleapis.com <span class="c"># Notice repo content available at /repo/ automatically</span> helm <span class="nb">install </span>grafana stable/grafana <span class="nt">-f</span> /repo/sample/grafana/with_lb.yaml <span class="nb">sleep </span>30 <span class="nb">export </span><span class="nv">SERVICE_IP</span><span class="o">=</span><span class="si">$(</span>kubectl get svc <span class="nt">--namespace</span> default grafana <span class="nt">-o</span> <span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{.status.loadBalancer.ingress[0].ip}'</span><span class="si">)</span> <span class="c"># Test if Grafana login page is reachable, exit code 1 means grep didn't found match</span> docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">--network</span> k3d-<span class="k">${</span><span class="nv">K3D_CLUSTER</span><span class="k">}</span> busybox /bin/sh <span class="nt">-c</span> <span class="s2">"wget </span><span class="k">${</span><span class="nv">SERVICE_IP</span><span class="k">}</span><span class="s2">:3000; cat index.html"</span> | <span class="nb">grep </span>Grafana</code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h4 id="expose-via-ingress">Expose via ingress</h4> <p>The last option is to expose containers using ingress. K3s/K3d is equipped with Traefik ingress controller, so the only missing part to start using it is to create ingress resource. In my case it is already handled by the Grafana chart, but it should be fairly easy to reproduce it. The way it works is not that different than load balancer way, there is just extra piece (Traefik) routing by URLs instead of ports. Notice adding the hostname in /etc/hosts of the container in below example.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">- k3d/k3d-run: step-name: Expose service via ingress <span class="nb">command</span>: | helm repo add stable https://kubernetes-charts.storage.googleapis.com <span class="c"># Notice repo content available at /repo/ automatically</span> helm <span class="nb">install </span>grafana stable/grafana <span class="nt">-f</span> /repo/sample/grafana/with_ingress.yaml <span class="nb">sleep </span>45 <span class="nb">export </span><span class="nv">INGRESS_IP</span><span class="o">=</span><span class="si">$(</span>kubectl get ing <span class="nt">--namespace</span> default grafana <span class="nt">-o</span> <span class="nv">jsonpath</span><span class="o">=</span><span class="s1">'{.status.loadBalancer.ingress[0].ip}'</span><span class="si">)</span> <span class="c"># Test if Grafana login page is reachable, exit code 1 means grep didn't found match</span> docker run <span class="nt">--rm</span> <span class="nt">-it</span> <span class="nt">--network</span> k3d-<span class="k">${</span><span class="nv">K3D_CLUSTER</span><span class="k">}</span> busybox <span class="se">\</span> /bin/sh <span class="nt">-c</span> <span class="s2">"echo '</span><span class="k">${</span><span class="nv">INGRESS_IP</span><span class="k">}</span><span class="s2"> chart-example.local' &gt;&gt; /etc/hosts; </span><span class="se">\</span><span class="s2"> wget http://chart-example.local/; cat index.html"</span> | <span class="nb">grep </span>Grafana</code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="publishing-your-own-orb">Publishing your own orb</h2> <p>As a sort of conclusion, I’d like to put together steps to publish your own orb. The docs are there but I felt a bit lost with all of those at the point where I knew what the content should be but not knowing how to make orb out of it.</p> <ol> <li>Write the steps as without using orb, as for any CI definition in .circleci/config.yml</li> <li>Move the generic orb commands to <a href="https://circleci.com/docs/2.0/orb-author/#writing-inline-orbs">inline orb</a> definition (still in your .circleci/config.yml). Notice that you might use the orb as it was already published as <code class="language-plaintext highlighter-rouge">k3d/&lt;command&gt;</code>. Test and fix.</li> </ol> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">version</span><span class="pi">:</span> <span class="m">2.1</span> <span class="na">orbs</span><span class="pi">:</span> <span class="na">python</span><span class="pi">:</span> <span class="s">circleci/python@0.3.2</span> <span class="na">k3d</span><span class="pi">:</span> <span class="na">commands</span><span class="pi">:</span> <span class="na">k3d-up</span><span class="pi">:</span> <span class="na">description</span><span class="pi">:</span> <span class="s">Starts k8s (k3d/k3s) cluster</span> <span class="na">parameters</span><span class="pi">:</span> <span class="na">cluster-name</span><span class="pi">:</span> <span class="na">description</span><span class="pi">:</span> <span class="s">Name for the k3d cluster</span> <span class="na">type</span><span class="pi">:</span> <span class="s">string</span> <span class="na">default</span><span class="pi">:</span> <span class="s">myk3d</span> <span class="s">(...)</span> <span class="na">jobs</span><span class="pi">:</span> <span class="na">test-on-k8s</span><span class="pi">:</span> <span class="na">executor</span><span class="pi">:</span> <span class="s">python/default</span> <span class="na">steps</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">setup_remote_docker</span> <span class="pi">-</span> <span class="s">checkout</span> <span class="pi">-</span> <span class="s">k3d/k3d-helpers</span> <span class="pi">-</span> <span class="na">k3d/k3d-up</span><span class="pi">:</span> <span class="na">cluster-name</span><span class="pi">:</span> <span class="s">circleci-k8s-1</span></code></pre></figure> <ol start="3"> <li>Install CircleCI CLI as described <a href="https://circleci.com/docs/2.0/creating-orbs/#step-1---set-up-the-circleci-cli">here</a> you would need CircleCI API token.</li> <li>Use <a href="https://github.com/CircleCI-Public/orb-starter-kit">orb-starter-kit</a> for templating a new repository with your orb. The repository name needs to match orb name, but you can change the repository name afterwards without problems, just update origin with something like: <code class="language-plaintext highlighter-rouge">git remote set-url origin https://hostname/USERNAME/REPOSITORY.git</code>.</li> <li>Migrate what you’ve done till now to new repository. You are no longer keeping things as yaml branches but rather as paths in src folder. The CI definition is in place, you are working on Alpha branch, which works as a staging/develop branch - pull requests into masters are triggering publishing the orb. So you are safe to just play with this branch until you are happy with the code. In my case I made all the examples also as integration tests, so that each commit triggers verification of the examples.</li> <li>Verify your orb against <a href="https://circleci.com/docs/2.0/orbs-best-practices/#orb-best-practices-guidelines">orb best practices </a></li> <li>Remember to update Readme.md, badges links and other parts that are still not fitted to your orb.</li> <li>You can use existing orb repos to see how it works i.e. <a href="https://github.com/CircleCI-Public/python-orb">python orb</a>.</li> <li>Create first PR into master, which will also publish the orb automatically. Just use <code class="language-plaintext highlighter-rouge">[semver:patch|minor|major|skip]</code> in name of the PR merge commit. So <code class="language-plaintext highlighter-rouge">[semver:patch]</code> would publish version 0.0.1, another commit with <code class="language-plaintext highlighter-rouge">[semver:patch]</code> will publish 0.0.2, <code class="language-plaintext highlighter-rouge">[semver:minor]</code> should publish 0.1.0 which you could treat as starting point, and so on and so forth.</li> <li>Your Alpha branch may look messy at this point, what you could do is just remove Alpha (after it was merged to master) and create new one out of master. Subsequent commits would probably have small changes or be PRs to Alpha.</li> </ol> <p>Other links:</p> <ul> <li><a href="https://circleci.com/docs/2.0/creating-orbs/">Publishing Orbs</a></li> <li><a href="https://circleci.com/blog/how-to-write-an-orb-that-people-will-love-to-use/">How to write an orb that people will love to use</a></li> </ul> <p><small><a href="#toc">Back to table of contents</a></small></p> https://devopsspiral.com/articles/linux/k3d-orb/ https://devopsspiral.com/articles/linux/ebpf-unlock/ Tue, 08 Sep 2020 00:00:00 +0000 <p>I heard “eBPF” so many times in recent days that I’ve decided to give it a try. I have very limited knowledge about kernel tracing so I thought it is good opportunity to learn something new. One particular <a href="https://youtu.be/16slh29iN1g">talk</a> (by Brendan Gregg) especially caught my attention and I recommend it if you want to get some general idea behind eBPF. At the beginning of his talk he is showing how he was able to monitor WiFi signal strength with eBPF. I seemed so easy that I wanted to mimic something similar. In this article I’m describing how I used eBPF tracing for locking and suspending my laptop using Bluetooth on my phone.</p> <div class="panel radius"> <p id="toc"><strong>Table of Contents</strong></p> <ul id="markdown-toc"> <li><a href="#so-what-is-really-ebpf" id="markdown-toc-so-what-is-really-ebpf">So what is really eBPF?</a></li> <li><a href="#bluetooth-case" id="markdown-toc-bluetooth-case">Bluetooth case</a></li> <li><a href="#into-the-ebpf" id="markdown-toc-into-the-ebpf">Into the eBPF</a></li> <li><a href="#system-lock-and-suspend-handling" id="markdown-toc-system-lock-and-suspend-handling">System lock and suspend handling</a></li> <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> </ul> </div> <h2 id="so-what-is-really-ebpf">So what is really eBPF?</h2> <p>There are many good sources to explain it along with the history of BPF (some of them you will find in this article), so I will try to make it simple.</p> <p>eBPF is a functionality of linux kernel that allows lightweight execution of user code as a response to kernel events. The events could be hardware/software events, tracing events both static (compiled into code) and dynamic (attached in runtime), etc. The code itself is limited in a sense that it is guaranteed to finish (no loops) and is verified before loading into kernel.</p> <p>Some facts and how others describes it:</p> <ul> <li>“Function-as-a-Service for kernel events” (Dan Wendlandt from his <a href="https://youtu.be/7PXQB-1U380">talk</a>)</li> <li>“Superpowers are coming to Linux” (Brendan Gregg from hist <a href="https://www.facebook.com/watch/?v=1693888610884236&amp;extid=gOZRPKYFuuClz71r">talk</a>)</li> <li>Alternative for service mesh <a href="https://youtu.be/Wocn6DK3FfM">Webinar: Comparing eBPF and Istio/Envoy for Monitoring Microservice Interactions</a></li> <li>Cilium which uses eBPF as a foundation, was recently announced GKE networking dataplane.</li> <li>kubectl has trace plugin for scheduling bpftrace programs</li> <li>Android has eBPF loader</li> </ul> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="bluetooth-case">Bluetooth case</h2> <p>At the beginning the only thing I knew is that I need to trace Bluetooth. But how should I know what are the function names that are called? Recalling some basic knowledge about linux kernel: kernel is the interface between user programs and hardware &gt; kernel uses kernel modules that are specific for particular hardware &gt; I should probably find Bluetooth module then. So by listing loaded modules I could find at least some anchor to begin with.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>lsmod Module Size Used by <span class="o">(</span>...<span class="o">)</span> bluetooth 573440 117 btrtl,btintel,btbcm,bnep,btusb,rfcomm</code></pre></figure> <p>Shockingly enough, the module I was looking for is named Bluetooth :). Turns out that with this information we can already limit tracing to specific module using trace-cmd. It is a wrapper around ftrace tracer, which in raw form requires writing values into filesystem. You can install it using something similar to <code class="language-plaintext highlighter-rouge">apt-get install -y trace-cmd</code>. Then you can actually use it for tracing functions of a specific module.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>trace-cmd record <span class="nt">-p</span> <span class="k">function</span> <span class="nt">-l</span> <span class="s1">':mod:bluetooth'</span> plugin <span class="s1">'function'</span> Hit Ctrl^C to stop recording ^CCPU0 data recorded at <span class="nv">offset</span><span class="o">=</span>0x671000 4096 bytes <span class="k">in </span>size CPU1 data recorded at <span class="nv">offset</span><span class="o">=</span>0x672000 4096 bytes <span class="k">in </span>size CPU2 data recorded at <span class="nv">offset</span><span class="o">=</span>0x673000 0 bytes <span class="k">in </span>size CPU3 data recorded at <span class="nv">offset</span><span class="o">=</span>0x673000 0 bytes <span class="k">in </span>size</code></pre></figure> <p>Seeing that there are non-zero bytes captured is always a good sign. To view the output you need to issue <em>report</em> command.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>trace-cmd report kworker/<span class="o">(</span>...<span class="o">)</span> 85164.256606: <span class="k">function</span>: process_adv_report kworker/<span class="o">(</span>...<span class="o">)</span> 85164.256607: <span class="k">function</span>: hci_find_irk_by_rpa kworker/<span class="o">(</span>...<span class="o">)</span> 85164.256608: <span class="k">function</span>: has_pending_adv_report kworker/<span class="o">(</span>...<span class="o">)</span> 85164.256609: <span class="k">function</span>: mgmt_device_found kworker/<span class="o">(</span>...<span class="o">)</span> 85164.256610: <span class="k">function</span>: hci_discovery_active kworker/<span class="o">(</span>...<span class="o">)</span> 85164.256612: <span class="k">function</span>: link_to_bdaddr.part.10</code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="into-the-ebpf">Into the eBPF</h2> <p>At this point I had a bunch of function calls, that I should use to somehow get the useful info about my phone’s Bluetooth state. You can find code of all the functions in kernel source code so in theory it should be matter of time to find the right one to trace. Well, it was pretty significant matter of time, spent on recalling C basics, HCI protocol events and bpftrace, which is what I wanted use in the end.</p> <p>So how the trial and error part looked like? I had to first install bpftrace from snap and overcome lockdown (see <a href="https://github.com/iovisor/bpftrace/issues/853">this issue</a>). You need to run all those commands as root too.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>snap <span class="nb">install </span>bpftrace <span class="nv">$ </span>snap connect bpftrace:system-trace</code></pre></figure> <p>I used kprobe, which is the kernel debugging mechanism that can be attached to function execution. Whenever function is entered you can access all its arguments (in bpftrace). So for example let’s say we want to trace <em>hci_cmd_complete_evt</em>, defined as:</p> <figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">static</span> <span class="kt">void</span> <span class="nf">hci_cmd_complete_evt</span><span class="p">(</span><span class="k">struct</span> <span class="n">hci_dev</span> <span class="o">*</span><span class="n">hdev</span><span class="p">,</span> <span class="k">struct</span> <span class="n">sk_buff</span> <span class="o">*</span><span class="n">skb</span><span class="p">,</span> <span class="n">u16</span> <span class="o">*</span><span class="n">opcode</span><span class="p">,</span> <span class="n">u8</span> <span class="o">*</span><span class="n">status</span><span class="p">,</span> <span class="n">hci_req_complete_t</span> <span class="o">*</span><span class="n">req_complete</span><span class="p">,</span> <span class="n">hci_req_complete_skb_t</span> <span class="o">*</span><span class="n">req_complete_skb</span><span class="p">)</span> <span class="p">{</span></code></pre></figure> <p>As you can see, there is hdev argument of type hci_dev which turns out to be structure holding a lot of information about the Bluetooth device. Let’s have a look just at the top fields of this struct.</p> <figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="k">struct</span> <span class="n">hci_dev</span> <span class="p">{</span> <span class="k">struct</span> <span class="n">list_head</span> <span class="n">list</span><span class="p">;</span> <span class="k">struct</span> <span class="n">mutex</span> <span class="n">lock</span><span class="p">;</span> <span class="kt">char</span> <span class="n">name</span><span class="p">[</span><span class="mi">8</span><span class="p">];</span></code></pre></figure> <p>Knowing all above, I could print the name of local device using below script.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># bt_dev_read.bt</span> 1 <span class="c">#!/snap/bin/bpftrace</span> 2 3 <span class="c">#include &lt;net/bluetooth/bluetooth.h&gt;</span> 4 <span class="c">#include &lt;net/bluetooth/hci_core.h&gt;</span> 5 6 7 kprobe:hci_cmd_complete_evt 8 <span class="o">{</span> 9 <span class="nv">$dev</span><span class="o">=(</span> struct hci_dev <span class="k">*</span><span class="o">)</span> arg0<span class="p">;</span> 10 <span class="nb">printf</span><span class="o">(</span><span class="s2">"%s</span><span class="se">\n</span><span class="s2">"</span>, <span class="nv">$dev</span>-&gt;name<span class="o">)</span><span class="p">;</span> 11 <span class="o">}</span></code></pre></figure> <p>So yes, bpftrace has its own language similar to C but consisting of only group of functions (see <a href="https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#bpftrace-reference-guide">reference guide</a>]). It is often used in a form of one-liners, but you can put it into file with shebang as I did.</p> <p>Going through the code, we need to first include kernel headers (lines 3-4) which is used to cast arg0 into hci_dev struct (line 9). In line 7, I’m attaching the code to kprobe:hci_cmd_complete_evt, which is eBPF at its beauty - you can execute your own code in kernel as a reaction to kernel event. In this case the reaction is just printing device name (line 10) every time <em>hci_cmd_complete_evt</em> function is entered. After running it you should see new line with device name every few seconds.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># bt_dev_read.bt</span> <span class="nv">$ </span>./bt_dev_read.bt Attaching 1 probe... hci0 hci0 hci0 hci0</code></pre></figure> <p>I had to also make sure that the target function is triggered frequently enough, so I found more usable function called <em>mgmt_device_found</em> that had all I need: MAC address (arg1) and RSSI (Received Signal Strength Indication - arg5).</p> <figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="kt">void</span> <span class="nf">mgmt_device_found</span><span class="p">(</span><span class="k">struct</span> <span class="n">hci_dev</span> <span class="o">*</span><span class="n">hdev</span><span class="p">,</span> <span class="n">bdaddr_t</span> <span class="o">*</span><span class="n">bdaddr</span><span class="p">,</span> <span class="n">u8</span> <span class="n">link_type</span><span class="p">,</span> <span class="n">u8</span> <span class="n">addr_type</span><span class="p">,</span> <span class="n">u8</span> <span class="o">*</span><span class="n">dev_class</span><span class="p">,</span> <span class="n">s8</span> <span class="n">rssi</span><span class="p">,</span> <span class="n">u32</span> <span class="n">flags</span><span class="p">,</span> <span class="n">u8</span> <span class="o">*</span><span class="n">eir</span><span class="p">,</span> <span class="n">u16</span> <span class="n">eir_len</span><span class="p">,</span> <span class="n">u8</span> <span class="o">*</span><span class="n">scan_rsp</span><span class="p">,</span> <span class="n">u8</span> <span class="n">scan_rsp_len</span><span class="p">)</span> <span class="p">{</span></code></pre></figure> <p>Even though bpftrace support array <em>[]</em> operator I had problems reading correctly bdaddr struct with its underlying array, so I end up using C trick of incrementing pointer to array directly. You can find more details in below snippet - I divided MAC into to halves and then put it back in printf. I’m also printing rssi after a space in same line.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># bt_rssi.bt</span> <span class="c">#!/snap/bin/bpftrace</span> kprobe:mgmt_device_found <span class="o">{</span> <span class="nv">$mac1</span><span class="o">=</span><span class="k">*</span><span class="o">(</span>arg1+3<span class="o">)</span> &amp; 0xffffff<span class="p">;</span> <span class="nv">$mac2</span><span class="o">=</span><span class="k">*</span><span class="o">(</span>arg1<span class="o">)</span> &amp; 0xffffff<span class="p">;</span> <span class="nb">printf</span><span class="o">(</span><span class="s2">"%X%X %d</span><span class="se">\n</span><span class="s2">"</span>, <span class="nv">$mac1</span>,<span class="nv">$mac2</span>, arg5<span class="o">)</span><span class="p">;</span> <span class="o">}</span></code></pre></figure> <p>The output from running this code looks as follows.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>./bt_rssi.bt Attaching 1 probe... 7AC8715201FD <span class="nt">-75</span> C2C62D1EC7 <span class="nt">-79</span> 30144A848783 <span class="nt">-87</span> C2C62D1EC7 <span class="nt">-82</span></code></pre></figure> <p>As a result I have per MAC signal strength that I can later on use to trigger lock and unlock of my Ubuntu system. That is great, but why not find disconnect function that would trigger system suspend whenever I disable Bluetooth on my phone. Fortunately it is there and its called <em>mgmt_device_disconnected</em> the bpftrace to output disconnected MAC can be found below.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># bt_disconnect.bt</span> <span class="c">#!/snap/bin/bpftrace</span> kprobe:mgmt_device_disconnected <span class="o">{</span> <span class="nv">$mac1</span><span class="o">=</span><span class="k">*</span><span class="o">(</span>arg1+3<span class="o">)</span> &amp; 0xffffff<span class="p">;</span> <span class="nv">$mac2</span><span class="o">=</span><span class="k">*</span><span class="o">(</span>arg1<span class="o">)</span> &amp; 0xffffff<span class="p">;</span> <span class="nb">printf</span><span class="o">(</span><span class="s2">"%X%X</span><span class="se">\n</span><span class="s2">"</span>, <span class="nv">$mac1</span>,<span class="nv">$mac2</span><span class="o">)</span><span class="p">;</span> <span class="o">}</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="system-lock-and-suspend-handling">System lock and suspend handling</h2> <p>We have now bt_rssi.bt showing MAC with corresponding signal power and bt_diconnect.bt showing MAC that just disconnected. It is time to shell-script it into actual system actions.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># bt_handler.sh</span> 1 <span class="c">#!/usr/bin/env bash</span> 2 <span class="nv">TARGET_MAC</span><span class="o">=</span><span class="nv">$1</span> 3 <span class="nv">LOCKFILE</span><span class="o">=</span>bt.lock 4 5 bt_lock<span class="o">()</span> <span class="o">{</span> 6 <span class="nv">args</span><span class="o">=(</span><span class="nv">$1</span><span class="o">)</span> 7 <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">args</span><span class="p">[0]</span><span class="k">}</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"</span><span class="nv">$TARGET_MAC</span><span class="s2">"</span> <span class="o">]</span> 8 <span class="k">then </span>9 <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">args</span><span class="p">[1]</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-lt</span> <span class="nt">-75</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="o">[</span> <span class="o">!</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$LOCKFILE</span><span class="s2">"</span> <span class="o">]</span> 10 <span class="k">then </span>11 <span class="o">&gt;</span><span class="s2">"</span><span class="nv">$LOCKFILE</span><span class="s2">"</span> 12 <span class="nb">echo</span> <span class="s2">"locked"</span> 13 loginctl lock-sessions 14 <span class="k">fi </span>15 <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">args</span><span class="p">[1]</span><span class="k">}</span><span class="s2">"</span> <span class="nt">-gt</span> <span class="nt">-65</span> <span class="o">]</span> <span class="o">&amp;&amp;</span> <span class="o">[</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$LOCKFILE</span><span class="s2">"</span> <span class="o">]</span> 16 <span class="k">then </span>17 <span class="nb">rm</span> <span class="nt">-f</span> <span class="s2">"</span><span class="nv">$LOCKFILE</span><span class="s2">"</span> 18 <span class="nb">echo</span> <span class="s2">"unlocked"</span> 19 loginctl unlock-sessions 20 <span class="k">fi </span>21 <span class="k">fi </span>22 <span class="o">}</span> 23 24 <span class="nb">export </span>TARGET_MAC 25 <span class="nb">export </span>LOCKFILE 26 <span class="nb">export</span> <span class="nt">-f</span> bt_lock 27 28 ./bt_disconnect.bt | <span class="nb">grep</span> <span class="nt">--line-buffered</span> <span class="s2">"</span><span class="nv">$TARGET_MAC</span><span class="s2">"</span> | xargs <span class="nt">-r</span> <span class="nt">-n1</span> bash <span class="nt">-c</span> <span class="s2">"&gt;</span><span class="nv">$LOCKFILE</span><span class="s2">; /bin/systemctl suspend"</span> &amp; 29 ./bt_rssi.bt | xargs <span class="nt">-r</span> <span class="nt">-n1</span> <span class="nt">-I</span> <span class="s1">'{}'</span> bash <span class="nt">-c</span> <span class="s1">'bt_lock "$@"'</span> _ <span class="o">{}</span></code></pre></figure> <p>The script takes only one argument - target MAC address (i.e. your phone Bluetooth MAC) and uses LOCKFILE file to mark if system is locked already. <em>bt_lock()</em> function parses the output of bt_rssi.bt to decide on locking or unlocking the session. Because xargs in line 29 passes one argument in form “&lt;MAC&gt; &lt;RSSI&gt;”, it needs to be broken into two arguments using bash array in line 6.</p> <p>Since xargs starts a separate process I had to export all the variables and function in lines 24-26. Last two lines of the script are just actual executions of bpftrace scripts.</p> <p>After running the whole script with Bluetooth working on both your pc and phone, you should see system being locked whenever you go away for ~2 meters and unlocked when phone is back near the pc. Turing off the Bluetooth on your phone will result in suspend. The extra feature is that when you turn on the Bluetooth after suspend and wake pc up it becomes unlocked, all of that without typing the password.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="conclusion">Conclusion</h2> <p>Just remember that the script showed in this article was created for educational purposes (just for fun) and it shouldn’t be used in places where you are exposed to anybody that could reach your keyboard. This is mainly because of the fact that your Bluetooth MAC becomes effectively password to your system, and despite of techniques that hides Bluetooth MAC it should be treated as public information. It works ok when using it at home, during pandemic, when the only hackers nearby are 2 and 6 years old and mostly using brute-force keyboard attacks, not Bluetooth MAC spoofing.</p> <p>As for eBPF, it is definitely very interesting concept that gains adoption in multiple fields. Vision behind it is far broader than tracing, performance and security cases. eBPF against current service mesh implementations with all its significant overhead is something worth looking at closely.</p> <p>You can find all the code I used here in <a href="https://github.com/devopsspiral/ebpf-bt-unlock">my git repository</a>.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> https://devopsspiral.com/articles/linux/ebpf-unlock/ https://devopsspiral.com/video/chatops-stackstorm/ Sun, 23 Aug 2020 00:00:00 +0000 https://devopsspiral.com/video/chatops-stackstorm/ https://devopsspiral.com/articles/k8s/testing-with-octopus/ Wed, 29 Jul 2020 00:00:00 +0000 <p>While looking for some tooling for testing Kubernetes deployments I’ve bumped into <a href="https://github.com/kyma-incubator/octopus">octopus</a> made as a part of Kyma project. It allows running tests in similar manner as helm test but treating test definitions and executions as kubernetes resources. I thought it is perfect tool for running tests that are encapsulated as containers, i.e. rf-service that I’m working on. In this article I’m describing how I made <a href="https://github.com/devopsspiral/KubeLibrary">KubeLibrary</a>, <a href="https://github.com/devopsspiral/rf-service/tree/v0.3.1">rf-service</a> and octopus work together.</p> <div class="panel radius"> <p id="toc"><strong>Table of Contents</strong></p> <ul id="markdown-toc"> <li><a href="#context" id="markdown-toc-context">Context</a></li> <li><a href="#why-octopus-is-such-a-good-fit" id="markdown-toc-why-octopus-is-such-a-good-fit">Why octopus is such a good fit?</a></li> <li><a href="#essential-changes-to-rf-service" id="markdown-toc-essential-changes-to-rf-service">Essential changes to rf-service</a> <ul> <li><a href="#cli-support" id="markdown-toc-cli-support">CLI support</a></li> <li><a href="#execution-by-tag" id="markdown-toc-execution-by-tag">Execution by tag</a></li> <li><a href="#dependency-resolution" id="markdown-toc-dependency-resolution">Dependency resolution</a></li> </ul> </li> <li><a href="#running-tests" id="markdown-toc-running-tests">Running tests</a></li> <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> </ul> </div> <h2 id="context">Context</h2> <p>This article is a part of series connected with testing on Kubernetes. You can find more info in following articles:</p> <p><a href="https://devopsspiral.com/articles/k8s/robotframework-kubelibrary/">Robot Framework library for testing Kubernetes</a> - in this part I’m describing Robot Framework library (Python) that uses Kubernetes client for getting info about your cluster and turning it into actual test suites.</p> <p><a href="https://devopsspiral.com/articles/k8s/robotframework-service/">Testing on kubernetes - rf-service</a> - this article describes Python service executed in a form of CronJob that actually runs the tests from KubeLibrary on kubernetes cluster.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="why-octopus-is-such-a-good-fit">Why octopus is such a good fit?</h2> <p>Testing kubernetes deployments is for sure something that is not that evolved as other parts of kubernetes world. We assume that things defined in YAML files will be delivered as we expect but does it really differ from regular programming? It becomes even more crucial when the final YAML is templated as in helm case. This was noticed by helm team and implemented as <a href="https://helm.sh/docs/topics/chart_tests/">helm test</a> functionality. The concept of encapsulating all the needed testing tools in container and running them against deployment is neat, the problem is that those tests are not treated as first class kubernetes citizens.</p> <p>Kyma-project Octopus took steps towards making this happen, you can find whole motivation in their <a href="https://kyma-project.io/blog/2020/1/16/integration-testing-in-k8s">blog post</a>. Basically, they turned 0/1 helm test approach into handling test as actual resource that can be retried, filtered and executed in parallel. It was used in integration testing context, but could be easily used as recurring health checks.</p> <p>Now you only need container with your test suites to execute on demand, which is exactly what rf-service is supposed to provide. Till now it was triggered as CronJob so there was schedule and possible repeatability, implemented changes allows taking advantage of running it within octopus. You can find more info about what was actually added in below section.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="essential-changes-to-rf-service">Essential changes to rf-service</h2> <p>In this article I’m only focusing on running rf-service as standalone container, but it can be started as REST API based service for running tests on demand.</p> <h3 id="cli-support">CLI support</h3> <p>Till now rf-service configuration was done by passing .json file with <em>fetcher</em> - functionality for collecting tests suites and <em>publisher</em> - functionality for publishing robotframework results. This meant that there was a need to make the container read some file or be mounted with it - by attaching ConfigMap for example. Running it using octopus demanded adding possibility to define all configuration using CLI parameters.</p> <p>To avoid maintaining growing list of hardcoded CLI parameters, they are generated dynamically from metadata of fetcher and publisher classes. Meaning that below CLI execution:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">rf-service <span class="nt">--LocalFetcher-src</span> ~/test/source <span class="nt">--LocalPublisher-dest</span> ~/test/results</code></pre></figure> <p>is equivalent with following json configuration file:</p> <figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w"> </span><span class="nl">"fetcher"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"LocalFetcher"</span><span class="p">,</span><span class="w"> </span><span class="nl">"src"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~/test/source"</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="nl">"publisher"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"LocalPublisher"</span><span class="p">,</span><span class="w"> </span><span class="nl">"dest"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~/test/results"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span></code></pre></figure> <p>allowing pretty flexible configuration of rf-service.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="execution-by-tag">Execution by tag</h3> <p>Tagging tests in Robot Framework is powerful way of handling test execution and in general categorizing testcases. Since rf-service fetches complete test suites, allowing executing only part of them is a must and has been mirrored from Robot Framework CLI. To include tags pass <code class="language-plaintext highlighter-rouge">-i &lt;tag&gt;</code> to exclude tags use <code class="language-plaintext highlighter-rouge">-e &lt;tag&gt;</code>. Those values are then passed to Robot Framework so you can expect all the behaviors to be exactly the same.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="dependency-resolution">Dependency resolution</h3> <p>In a path towards making rf-service generic enough to be executed as a base for different kinds of testcases, support for pip requirements was added. This way if fetcher collects directory containing <em>requirements.txt</em> file, it will install packages as with <code class="language-plaintext highlighter-rouge">pip install -r requirements.txt</code>. Just remember first spotted requirements.txt file will be used, so it is best to keep one in top level directory.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="running-tests">Running tests</h2> <p>Alright let’s run some tests then. I will be running testcases from <a href="https://github.com/devopsspiral/KubeLibrary/tree/v0.1.4/testcases">KubeLibrary/testcases</a> and using KubeLibrary as a testing library itself.</p> <p>As a first step you need to install octopus from the chart provided in its <a href="https://github.com/kyma-incubator/octopus">repository</a>.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone https://github.com/kyma-incubator/octopus <span class="nb">cd </span>octopus helm <span class="nb">install </span>octopus ./chart/octopus/</code></pre></figure> <p>Then, we need to define what and how the tests will be executed. All those things are configured using kubernetes resources (CRD) brought by octopus. In <em>TestDefinition</em> you define how the test container is executed in similar way kubernetes deployment is defined. In my case, test definition is defined as follows:</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># test-definition.yaml</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">testing.kyma-project.io/v1alpha1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">TestDefinition</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">component</span><span class="pi">:</span> <span class="s">kube-tests</span> <span class="na">name</span><span class="pi">:</span> <span class="s">test-example</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">template</span><span class="pi">:</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">serviceAccountName</span><span class="pi">:</span> <span class="s">octopus-sa</span> <span class="na">containers</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">test</span> <span class="na">image</span><span class="pi">:</span> <span class="s">mwcislo/rf-service:0.3.0</span> <span class="na">command</span><span class="pi">:</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">rf-service"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">--ZipFetcher-url"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">https://github.com/devopsspiral/KubeLibrary/archive/v0.1.4.zip"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">--ZipFetcher-path"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">KubeLibrary-0.1.4/testcases"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">--LocalPublisher-dest"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">somecontext"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">-i"</span> <span class="pi">-</span> <span class="s2">"</span><span class="s">octopus"</span> <span class="na">env</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_POD_PATTERN</span> <span class="na">value</span><span class="pi">:</span> <span class="s">octopus.*</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_POD_NAMESPACE</span> <span class="na">value</span><span class="pi">:</span> <span class="s">default</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_POD_LABELS</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{</span><span class="se">\"</span><span class="s">app</span><span class="se">\"</span><span class="s">:</span><span class="se">\"</span><span class="s">octopus</span><span class="se">\"</span><span class="s">}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_ENV_VARS</span> <span class="na">value</span><span class="pi">:</span> <span class="s2">"</span><span class="s">{</span><span class="se">\"</span><span class="s">SECRET_NAME</span><span class="se">\"</span><span class="s">:</span><span class="se">\"</span><span class="s">webhook-server-secret</span><span class="se">\"</span><span class="s">}"</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_RESOURCE_REQUESTS_CPU</span> <span class="na">value</span><span class="pi">:</span> <span class="s">100m</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_RESOURCE_REQUESTS_MEMORY</span> <span class="na">value</span><span class="pi">:</span> <span class="s">20Mi</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_RESOURCE_LIMITS_CPU</span> <span class="na">value</span><span class="pi">:</span> <span class="s">100m</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">KLIB_RESOURCE_LIMITS_MEMORY</span> <span class="na">value</span><span class="pi">:</span> <span class="s">30Mi</span></code></pre></figure> <p>Shortly speaking I’m telling rf-service to take tests from KubeLibrary repository (branch master) on path testcases/ and publish results locally in directory <em>somecontext</em> - the actual path doesn’t really matter in this case. The execution should only include tests with tag <em>octopus</em>. All the environment variables are just parameterized test variables, it is attempt to make the tests more generic and possibly use same tests with different test targets, i.e. different services.</p> <p>You might noticed <code class="language-plaintext highlighter-rouge">serviceAccountName: octopus-sa</code> part, this is needed to run the test container in privileged mode so that it can talk to different parts of kubernetes API and depends on what you really need to test. Below I’m showing the resources that needs to be applied to make it cluster-admin privileged.</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># serviceaccount.yaml</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">octopus-sa</span> <span class="nn">---</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io/v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRoleBinding</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">octopus-admin-binding</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span> <span class="na">subjects</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ServiceAccount</span> <span class="na">name</span><span class="pi">:</span> <span class="s">octopus-sa</span> <span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span> <span class="na">roleRef</span><span class="pi">:</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterRole</span> <span class="na">name</span><span class="pi">:</span> <span class="s">cluster-admin</span> <span class="na">apiGroup</span><span class="pi">:</span> <span class="s">rbac.authorization.k8s.io</span></code></pre></figure> <p>The second part that needs to be applied is <em>ClusterTestSuite</em>, which represents how the test should be actually executed. Tests are matched by label, which again allows convenient way of grouping tests. Other parameters are <strong>count</strong> - how many times test should be repeated, <strong>maxRetries</strong> - how many times can be repeated if failed and <strong>concurrency</strong> - how many tests can run in parallel, in our case we only have one test so any changes here takes no effect, but when you are running a lot of tests (TestsDefinitions) it can save a lot of time.</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="c1"># cluster-test-suite.yaml</span> <span class="na">apiVersion</span><span class="pi">:</span> <span class="s">testing.kyma-project.io/v1alpha1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterTestSuite</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">controller-tools.k8s.io</span><span class="pi">:</span> <span class="s2">"</span><span class="s">1.0"</span> <span class="na">name</span><span class="pi">:</span> <span class="s">testsuite-selected-by-labels</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">count</span><span class="pi">:</span> <span class="m">1</span> <span class="na">maxRetries</span><span class="pi">:</span> <span class="m">1</span> <span class="na">concurrency</span><span class="pi">:</span> <span class="m">1</span> <span class="na">selectors</span><span class="pi">:</span> <span class="na">matchLabelExpressions</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">component=kube-tests</span></code></pre></figure> <p>To run the tests just apply all the files using kubectl, starting with serviceaccount.yaml. After couple of seconds you should see new pod being created and inside of it, there will be rf-service running KubeLibrary tests from within the cluster. You should see all the results in container logs, to view them just use regular logs lookup <code class="language-plaintext highlighter-rouge">kubectl logs oct-tp-testsuite-selected-by-labels-test-example-0</code>:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">(</span>...<span class="o">)</span> <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9 <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master.Testcases <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master.Testcases.Pod <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master.Testcases.Pod.Pod <span class="o">==============================================================================</span> Pod images has correct version | PASS | <span class="nt">------------------------------------------------------------------------------</span> Pod has enough replicas | PASS | <span class="nt">------------------------------------------------------------------------------</span> Pod has not been restarted | PASS | <span class="nt">------------------------------------------------------------------------------</span> Pod have correct labels | PASS | <span class="nt">------------------------------------------------------------------------------</span> Pod has correct limits/requests | PASS | <span class="nt">------------------------------------------------------------------------------</span> Pod has correct <span class="nb">env </span>variables | PASS | <span class="nt">------------------------------------------------------------------------------</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master.Testcases.... | PASS | 6 critical tests, 6 passed, 0 failed 6 tests total, 6 passed, 0 failed <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master.Testcases.Pod | PASS | 6 critical tests, 6 passed, 0 failed 6 tests total, 6 passed, 0 failed <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master.Testcases | PASS | 6 critical tests, 6 passed, 0 failed 6 tests total, 6 passed, 0 failed <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9.KubeLibrary-master | PASS | 6 critical tests, 6 passed, 0 failed 6 tests total, 6 passed, 0 failed <span class="o">==============================================================================</span> A6D70814-B63C-49Ba-B82F-8778B7D86De9 | PASS | 6 critical tests, 6 passed, 0 failed 6 tests total, 6 passed, 0 failed <span class="o">==============================================================================</span> Output: /output.xml</code></pre></figure> <p>Test execution status is kept in ClusterTestSuite resource, you can use it to view resulting status, start time, completion time and others. This allows getting overview of multiple suites execution and keeping its records.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$kubectl</span> get ClusterTestSuite NAME AGE testsuite-selected-by-labels 18s <span class="nv">$kubectl</span> describe ClusterTestSuite testsuite-selected-by-labels Name: testsuite-selected-by-labels Namespace: Labels: controller-tools.k8s.io<span class="o">=</span>1.0 Annotations: API Version: testing.kyma-project.io/v1alpha1 Kind: ClusterTestSuite Metadata: Creation Timestamp: 2020-07-26T13:21:14Z Generation: 1 Resource Version: 684 Self Link: /apis/testing.kyma-project.io/v1alpha1/clustertestsuites/testsuite-selected-by-labels UID: f2f11a7d-8927-47cb-a125-0f107018194c Spec: Concurrency: 1 Count: 1 Max Retries: 1 Selectors: Match Label Expressions: <span class="nv">component</span><span class="o">=</span>kube-tests Status: Completion Time: 2020-07-26T13:22:16Z Conditions: Status: False Type: Running Status: True Type: Succeeded Results: Executions: Completion Time: 2020-07-26T13:22:16Z Id: oct-tp-testsuite-selected-by-labels-test-example-0 Pod Phase: Succeeded Start Time: 2020-07-26T13:21:15Z Name: test-example Namespace: default Status: Succeeded Start Time: 2020-07-26T13:21:14Z Events: &lt;none&gt;</code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="conclusion">Conclusion</h2> <p>In my opinion octopus is very interesting project and might play important role whenever kubernetes is used, by filling testing gap. As someone said YAML became k8s programming language and as every programming language needs some holistic approach to testing. Observing what octopus can and what are the proposals seen in the project <a href="https://github.com/kyma-incubator/octopus/issues">issues</a> seems it can be good move towards it.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> https://devopsspiral.com/articles/k8s/testing-with-octopus/ https://devopsspiral.com/video/chatops-intro/ Tue, 12 May 2020 00:00:00 +0000 https://devopsspiral.com/video/chatops-intro/ https://devopsspiral.com/articles/k8s/robotframework-service-fe/ Fri, 10 Apr 2020 00:00:00 +0000 <p>Vue.js is getting more and more popularity, it is still behind React.js and Angular but catching up. It is known for its clarity and flexibility, that is why it makes a perfect opportunity to learn for frontend noobs such as myself.</p> <div class="panel radius"> <p id="toc"><strong>Table of Contents</strong></p> <ul id="markdown-toc"> <li><a href="#context" id="markdown-toc-context">Context</a></li> <li><a href="#why-vuejs-" id="markdown-toc-why-vuejs-">Why Vue.js ?</a> <ul> <li><a href="#installation-and-first-project" id="markdown-toc-installation-and-first-project">Installation and first project</a></li> <li><a href="#components" id="markdown-toc-components">Components</a> <ul> <li><a href="#rendering" id="markdown-toc-rendering">Rendering</a></li> <li><a href="#runnervue-and-recursive-components" id="markdown-toc-runnervue-and-recursive-components">Runner.vue and recursive components</a></li> </ul> </li> <li><a href="#routing" id="markdown-toc-routing">Routing</a></li> <li><a href="#dockerizing" id="markdown-toc-dockerizing">Dockerizing</a></li> </ul> </li> <li><a href="#changes-in-rf-service" id="markdown-toc-changes-in-rf-service">Changes in rf-service</a> <ul> <li><a href="#api" id="markdown-toc-api">API</a></li> <li><a href="#changes-in-helm" id="markdown-toc-changes-in-helm">Changes in helm</a></li> </ul> </li> <li><a href="#testing" id="markdown-toc-testing">Testing</a></li> <li><a href="#conclusion" id="markdown-toc-conclusion">Conclusion</a></li> </ul> </div> <h2 id="context">Context</h2> <p>This article is a part of series connected with testing on Kubernetes. It is not mandatory to go through previous articles, but it gives a better understanding of what I’m actually trying to achieve here. The Vue.js intro part is written independently so that with below short description of each of the previous parts you should be fine with going through it.</p> <p><a href="https://devopsspiral.com/articles/k8s/robotframework-kubelibrary/">Robot Framework library for testing Kubernetes</a> - in this part I’m describing Robot Framework library (Python) that uses Kubernetes client for getting info about your cluster and turning it into actual test suites.</p> <p><a href="https://devopsspiral.com/articles/k8s/robotframework-service/">Testing on kubernetes - rf-service</a> - this article describes Python service executed in a form of CronJob that actually runs the tests from KubeLibrary on kubernetes cluster.</p> <p>The article your reading is a next step, where rf-service gets api and frontend so that tests can be executed on demand and results can be viewed in single web UI.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="why-vuejs-">Why Vue.js ?</h2> <p>Instead of repeating same story of what Vue.js is, I would rather focus why it seemed perfect for me - knowing only basics of HTML, js and CSS. Vue.js is said to leverage best parts from React and Angular, making it also more lightweight and easy to learn. This was most important for me so that I can learn quickly and also not spend time on something that is far from the main players. You can find more detailed comparison created by Vue.js core team - <a href="https://vuejs.org/v2/guide/comparison.html">Comparison with Other Frameworks</a>.</p> <p>Vue.js is also in upward trend which makes it perfect competence investment. On top of that you can expect many other people facing similar problems while learning it so access to information and learning should not be a problem. More on that in <a href="https://medium.com/@inverita/why-is-vue-js-a-front-end-trend-of-2020-4081e8c799aa">Why Is Vue.js a Front-end Trend of 2020?</a>.</p> <p>The third point that really caught my attention is that you can write mobile apps using Vue Native. I didn’t tried it yet, but possibility to implement some cool projects on your mobile sounds very tempting.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="installation-and-first-project">Installation and first project</h3> <p>The recommended method for installation is through <em>npm</em>. If you don’t have it yet, this is the prerequisite for further steps. You can install Vue with below command:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">npm <span class="nb">install </span>vue</code></pre></figure> <p>Installing vue-cli makes a lot of things easier, so I used it also. You can add it with:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">npm <span class="nb">install</span> <span class="nt">-g</span> @vue/cli</code></pre></figure> <p>You can create your first project by executing:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">vue create &lt;project-name&gt;</code></pre></figure> <p>This will create everything you need to see Vue.js in action including structure and example project that can be base for you changes as in my case. To see the UI just go to project folder and start the development server with:</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">npm run serve</code></pre></figure> <p>You should see Hello world app on <em>http://localhost:8080/</em>.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="components">Components</h3> <p>Simplifying things a lot we can say that Vue.js is templating HTMLs documents using JavaScript. Because this is exactly what browser engines works on and understands. Having HTML document, browser can build DOM (Document Object Model) which is the logical representation of HTML text and this is what is then rendered in front of user’s eyes. Modern frameworks uses Virtual DOM which is the higher level abstraction that allows more efficient and manageable way of providing content representation.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h4 id="rendering">Rendering</h4> <p>Knowing above let’s just track how actually Vue.js code is being placed in final HTML document. Before that, we can already switch to the actual code of rf-service frontend. Just clone the repo from <a href="https://github.com/devopsspiral/rf-service-fe">devopsspiral/rf-service-fe</a>. Again it is vue project created with vue cli.</p> <p>We will follow top-down path. If you go to <em>public/index.html</em>, you will see how the Vue.js app is mounted.</p> <figure class="highlight"><pre><code class="language-html" data-lang="html"># public/index.html <span class="nt">&lt;div</span> <span class="na">id=</span><span class="s">"app"</span><span class="nt">&gt;&lt;/div&gt;</span></code></pre></figure> <p>This is the place where <em>src/App.vue</em> is referenced. In App.vue there is <em>template</em> and <em>style</em> part, so you just literally create parts of your app as self-contained source of HTML and CSS. This looks really simple and is easy to manage.</p> <div class="alert-box text radius "><p>Creating .vue files with template, style and js code is not the only way for templating used in Vue. At some point you can find that it is not enough for all the cases, you can find more info <a href="https://vuejsdevelopers.com/2017/03/24/vue-js-component-templates/">here</a>.</p> </div> <figure class="highlight"><pre><code class="language-html" data-lang="html"># src/App.vue <span class="nt">&lt;template&gt;</span> <span class="nt">&lt;div</span> <span class="na">id=</span><span class="s">"app"</span><span class="nt">&gt;</span> <span class="nt">&lt;div</span> <span class="na">id=</span><span class="s">"nav"</span><span class="nt">&gt;</span> <span class="nt">&lt;router-link</span> <span class="na">to=</span><span class="s">"/"</span><span class="nt">&gt;</span>Runner<span class="nt">&lt;/router-link&gt;</span> | <span class="nt">&lt;router-link</span> <span class="na">to=</span><span class="s">"/results"</span><span class="nt">&gt;</span>Results<span class="nt">&lt;/router-link&gt;</span> | <span class="nt">&lt;router-link</span> <span class="na">to=</span><span class="s">"/config"</span><span class="nt">&gt;</span>Configure<span class="nt">&lt;/router-link&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;router-view/&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;/template&gt;</span></code></pre></figure> <p>I will explain routers later on, for now we can just assume that <em>&lt;router-link&gt;</em> is just a link to other parts of our application. Those can be found in <em>src/views</em> and <em>src/components</em>. There is not much difference of how you create views and components, they are kept separate just for clarity. They serve different functions though. Views are used to present parts of your app to the user, while components could be some internal machinery of your app referenced in views.</p> <p>In my case Runner.vue, Result.vue and Config.vue are kept as views in <em>src/views</em> directory. If you look inside those files, you will see that apart from template and style, we have now extra vue/js logic that makes the whole thing dynamic.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h4 id="runnervue-and-recursive-components">Runner.vue and recursive components</h4> <p>I will just focus on Runner.vue as other views are pretty similar. The template part is simple, we have two components: button and test-list. The first one is just regular HTML button but with vue attribute <em>v-on:click</em>, that is pointing into function in <em>script</em> part in <em>method</em> section. This is how you define action from JS to be linked with HTML object. Just to better understand what is done there I’m performing a POST http call to rf-service api using Axios.</p> <figure class="highlight"><pre><code class="language-html" data-lang="html"># src/views/Runner.vue <span class="nt">&lt;template&gt;</span> <span class="nt">&lt;div&gt;</span> <span class="nt">&lt;button</span> <span class="na">v-on:click=</span><span class="s">"runTests"</span><span class="nt">&gt;</span>Run All<span class="nt">&lt;/button&gt;</span> <span class="nt">&lt;div</span> <span class="na">id=</span><span class="s">"tests"</span><span class="nt">&gt;&lt;test-list</span> <span class="na">:children=</span><span class="s">"tests"</span> <span class="na">:indentation=</span><span class="s">"0"</span><span class="nt">&gt;&lt;/test-list&gt;&lt;/div&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;/template&gt;</span></code></pre></figure> <p>The second object in template is test-list which is as component in <em>src/components/TestList.vue</em>. As you can see Vue components are not only modules in Vue.js code, they become actual HTML tags that can be used in templates. Test-list is a bit more interesting because it implements test tree structure using recursion. If you look into its definition it uses recursive call in template section.</p> <figure class="highlight"><pre><code class="language-html" data-lang="html"># src/components/TestList.vue <span class="nt">&lt;template&gt;</span> <span class="nt">&lt;div&gt;</span> <span class="nt">&lt;div</span> <span class="na">:style=</span><span class="s">"indent"</span> <span class="err">@</span><span class="na">click=</span><span class="s">"toggleChildren"</span><span class="nt">&gt;&lt;/div&gt;</span> <span class="nt">&lt;div</span> <span class="na">v-if=</span><span class="s">"showChildren"</span><span class="nt">&gt;</span> <span class="c">&lt;!-- Recursive call --&gt;</span> <span class="nt">&lt;test-list</span> <span class="na">v-for=</span><span class="s">"child in children"</span> <span class="na">:children=</span><span class="s">"child.children"</span> <span class="na">:name=</span><span class="s">"child.name"</span> <span class="na">:key=</span><span class="s">"child.name"</span> <span class="na">:indentation=</span><span class="s">"indentation + 1"</span> <span class="nt">&gt;</span> <span class="nt">&lt;/test-list&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;/div&gt;</span> <span class="nt">&lt;/template&gt;</span></code></pre></figure> <p>Other than that, there is <code class="language-plaintext highlighter-rouge">@click</code> which is shorthand for <code class="language-plaintext highlighter-rouge">v-on:click</code> and <code class="language-plaintext highlighter-rouge">:style</code> which is shorthand for <code class="language-plaintext highlighter-rouge">v-bind:style</code> - this is probably the most used vue directive. It binds the value of the parameter with variable or function in JS and this is where whole reactive magic happens. You don’t need to care about updating anything, whatever changes in JS is reflected in HTML.</p> <p>You can also find <em>v-if</em> and <em>v-for</em> directives that basically works similar to what can be found in other templating engines (like jinja) and represents conditional rendering and loops. In this case v-if will conditionally render &lt;div&gt;, which is by the way boundary of v-if - you don’t use anything similar to <em>endif</em> for example. On the other side v-for will multiply test-list tags for each child in children.</p> <figure class="highlight"><pre><code class="language-html" data-lang="html"># src/components/TestList.vue <span class="nt">&lt;script&gt;</span> <span class="k">export</span> <span class="k">default</span> <span class="p">{</span> <span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">test-list</span><span class="dl">'</span><span class="p">,</span> <span class="na">props</span><span class="p">:</span> <span class="p">[</span><span class="dl">'</span><span class="s1">name</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">children</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">indentation</span><span class="dl">'</span><span class="p">],</span> <span class="nx">data</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="p">{</span> <span class="na">tests</span><span class="p">:</span> <span class="p">{},</span> <span class="na">showChildren</span><span class="p">:</span> <span class="kc">true</span> <span class="p">};</span> <span class="p">},</span> <span class="na">computed</span><span class="p">:</span> <span class="p">{</span> <span class="nx">indent</span><span class="p">()</span> <span class="p">{</span> <span class="k">return</span> <span class="p">{</span> <span class="na">transform</span><span class="p">:</span> <span class="s2">`translate(</span><span class="p">${</span><span class="k">this</span><span class="p">.</span><span class="nx">indentation</span> <span class="o">*</span> <span class="mi">50</span><span class="p">}</span><span class="s2">px)`</span><span class="p">}</span> <span class="p">}</span> <span class="p">},</span> <span class="na">methods</span><span class="p">:</span> <span class="p">{</span> <span class="nx">toggleChildren</span><span class="p">()</span> <span class="p">{</span> <span class="k">this</span><span class="p">.</span><span class="nx">showChildren</span> <span class="o">=</span> <span class="o">!</span><span class="k">this</span><span class="p">.</span><span class="nx">showChildren</span><span class="p">;</span> <span class="p">},</span> <span class="p">}</span> <span class="p">}</span> <span class="nt">&lt;/script&gt;</span></code></pre></figure> <p>Going to the script section, there are couple of distinct parts there. <em>name</em> and <em>props</em> is defining test-list representation as (HTML) component with parameters. <code class="language-plaintext highlighter-rouge">data()</code> is the place for all variables available across whole file. Just remember to add <code class="language-plaintext highlighter-rouge">this.</code> if you are referencing this variable inside script section. And lastly, we have <em>computed</em> and <em>methods</em> section. You may wander what is the difference if all in all they are both functions. The difference is in how they are used. <code class="language-plaintext highlighter-rouge">indent()</code> is actually used as value not action as in <code class="language-plaintext highlighter-rouge">toggleChildren()</code> case. This is why Vue needs to take some extra care of this value in a sense of reactivity i.e. compute the new value whenever <code class="language-plaintext highlighter-rouge">this.indentation</code> changes.</p> <p>In Runner.vue there is another interesting section called <em>created</em> which is executed on page loading. In this case, it is another call to api to fetch some parameters metadata.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="routing">Routing</h3> <p>We did a little round tour of Vue elements and we’re back to the point where we would like to put things together. This is where routers come into play. As in other programming languages routers directs the flow into classes or methods, in Vue.js those will be components. If you have your app already created without router, don’t worry you can always add it using vue cli.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">vue add router</code></pre></figure> <p>This will create necessary files in your project and you will be ready to use it. The actual routing paths are kept in <em>src/router/index.js</em> in routes variable. This is where you link paths with components.</p> <figure class="highlight"><pre><code class="language-js" data-lang="js"><span class="err">#</span> <span class="nx">src</span><span class="o">/</span><span class="nx">router</span><span class="o">/</span><span class="nx">index</span><span class="p">.</span><span class="nx">js</span> <span class="kd">const</span> <span class="nx">routes</span> <span class="o">=</span> <span class="p">[</span> <span class="p">{</span> <span class="na">path</span><span class="p">:</span> <span class="dl">'</span><span class="s1">/</span><span class="dl">'</span><span class="p">,</span> <span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Runner</span><span class="dl">'</span><span class="p">,</span> <span class="na">component</span><span class="p">:</span> <span class="nx">Runner</span> <span class="p">},</span> <span class="p">{</span> <span class="na">path</span><span class="p">:</span> <span class="dl">'</span><span class="s1">/config</span><span class="dl">'</span><span class="p">,</span> <span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Config</span><span class="dl">'</span><span class="p">,</span> <span class="na">component</span><span class="p">:</span> <span class="nx">Config</span> <span class="p">},</span> <span class="p">{</span> <span class="na">path</span><span class="p">:</span> <span class="dl">'</span><span class="s1">/results</span><span class="dl">'</span><span class="p">,</span> <span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Results</span><span class="dl">'</span><span class="p">,</span> <span class="na">component</span><span class="p">:</span> <span class="nx">Results</span> <span class="p">},</span> <span class="p">]</span></code></pre></figure> <p>Having the paths you can now link to them as it was done in App.vue.</p> <figure class="highlight"><pre><code class="language-html" data-lang="html"># src/App.vue <span class="nt">&lt;router-link</span> <span class="na">to=</span><span class="s">"/"</span><span class="nt">&gt;</span>Runner<span class="nt">&lt;/router-link&gt;</span> | <span class="nt">&lt;router-link</span> <span class="na">to=</span><span class="s">"/results"</span><span class="nt">&gt;</span>Results<span class="nt">&lt;/router-link&gt;</span> | <span class="nt">&lt;router-link</span> <span class="na">to=</span><span class="s">"/config"</span><span class="nt">&gt;</span>Configure<span class="nt">&lt;/router-link&gt;</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="dockerizing">Dockerizing</h3> <p>All right, we have our app running well in development, what if we would like to actually share it with others as a dockerized service? This couldn’t be simpler, just use Dockerfile in the repo (taken from <a href="https://vuejs.org/v2/cookbook/dockerize-vuejs-app.html">https://vuejs.org/v2/cookbook/dockerize-vuejs-app.html</a>). It doesn’t have any app specific parts so it can be used as a base for whatever app you are building.</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># Dockerfile</span> FROM node:lts-alpine <span class="c"># install simple http server for serving static content</span> RUN npm <span class="nb">install</span> <span class="nt">-g</span> http-server <span class="c"># make the 'app' folder the current working directory</span> WORKDIR /app <span class="c"># copy both 'package.json' and 'package-lock.json' (if available)</span> COPY package<span class="k">*</span>.json ./ <span class="c"># install project dependencies</span> RUN npm <span class="nb">install</span> <span class="c"># copy project files and folders to the current working directory (i.e. 'app' folder)</span> COPY <span class="nb">.</span> <span class="nb">.</span> <span class="c"># build app for production with minification</span> RUN npm run build EXPOSE 8080 CMD <span class="o">[</span> <span class="s2">"http-server"</span>, <span class="s2">"dist"</span> <span class="o">]</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="changes-in-rf-service">Changes in rf-service</h2> <p>Getting back to broader context of this article, <a href="https://github.com/devopsspiral/rf-service">rf-service</a> got api and frontend. It is still possible to run tests from cron job, but the results can be viewed in Results tab in rf-service-fe. Switch between modes is done by passing <code class="language-plaintext highlighter-rouge">.Values.config</code> helm parameter with configuration of fetcher and publisher as before. By default it is defined as empty string and all the configuration is done using Vue.js frontend.</p> <p>As a result we have now 3 containers running: frontend with Vue.js app, rf-service and its api and Caddy serving as a store for Robot Framework results.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="api">API</h3> <p>rf-service has now api with following endpoints</p> <table> <thead> <tr> <th>Path</th> <th>Method</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>/api/publishers</td> <td>GET</td> <td>list of available Publishers</td> </tr> <tr> <td>/api/publishers_conf</td> <td>POST, GET</td> <td>configure Publisher</td> </tr> <tr> <td>/api/fetchers</td> <td>GET</td> <td>list of available Fetchers</td> </tr> <tr> <td>/api/fetchers_conf</td> <td>POST, GET</td> <td>configure Fetcher</td> </tr> <tr> <td>/api/tests</td> <td>GET</td> <td>view loaded tests</td> </tr> <tr> <td>/api/run</td> <td>POST</td> <td>run all the tests</td> </tr> </tbody> </table> <p>All the paths have explicit /api/ prefix for simplicity, this could be replaced by Flask blueprints or simply by mounting the app on specific path. Api is also missing swagger docs, which might be covered in future.</p> <p>The app in final container is served by gevent which allowed to expose api without extra non-python dependencies and allowed easy implementation of backward compatibility.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> <h3 id="changes-in-helm">Changes in helm</h3> <p>As a result of having 3 containers in place. I decided to put them into one pod, this resulted in multiple ports in one service</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span> <span class="na">metadata</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s">-rf-service</span> <span class="na">labels</span><span class="pi">:</span> <span class="na">spec</span><span class="pi">:</span> <span class="na">type</span><span class="pi">:</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="na">targetPort</span><span class="pi">:</span> <span class="s">http</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> <span class="na">name</span><span class="pi">:</span> <span class="s">http</span> <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">8090</span> <span class="na">targetPort</span><span class="pi">:</span> <span class="s">caddy</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> <span class="na">name</span><span class="pi">:</span> <span class="s">caddy</span> <span class="pi">-</span> <span class="na">port</span><span class="pi">:</span> <span class="m">5000</span> <span class="na">targetPort</span><span class="pi">:</span> <span class="s">api</span> <span class="na">protocol</span><span class="pi">:</span> <span class="s">TCP</span> <span class="na">name</span><span class="pi">:</span> <span class="s">api</span> <span class="na">selector</span><span class="pi">:</span></code></pre></figure> <p>and ingress. Expose over ingress became mandatory to allows easy handling on frontend side with paths /caddy/ and /api/. This is exactly why rf-service api needs to have /api/ prefix. Caddy /caddy/ path is handled by using rewrite (see <code class="language-plaintext highlighter-rouge">.Values.caddy.setup</code> and snippet below).</p> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"> <span class="na">rules</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">host</span><span class="pi">:</span> <span class="na">http</span><span class="pi">:</span> <span class="na">paths</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/"</span> <span class="na">backend</span><span class="pi">:</span> <span class="na">serviceName</span><span class="pi">:</span> <span class="na">servicePort</span><span class="pi">:</span> <span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/caddy/"</span> <span class="na">backend</span><span class="pi">:</span> <span class="na">serviceName</span><span class="pi">:</span> <span class="na">servicePort</span><span class="pi">:</span> <span class="m">8090</span> <span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/api/"</span> <span class="na">backend</span><span class="pi">:</span> <span class="na">serviceName</span><span class="pi">:</span> <span class="na">servicePort</span><span class="pi">:</span> <span class="s">5000</span></code></pre></figure> <figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"> <span class="c1"># Bind address</span> <span class="s">:8090</span> <span class="s">tls off</span> <span class="s">log stdout</span> <span class="s">errors stderr</span> <span class="s"># After this line, all other paths are relative to root.</span> <span class="s">root /tmp/store</span> <span class="s">browse /</span> <span class="s">rewrite /caddy/ {</span> <span class="s">to /{file} /</span> <span class="s">}</span> <span class="s">upload /uploads {</span> <span class="s">to "/tmp/store"</span> <span class="s">}</span></code></pre></figure> <p><small><a href="#toc">Back to table of contents</a></small></p> <h2 id="testing">Testing</h2> <p>You can test the whole setup by first installing the chart on your k8s cluster (k3d in my case):</p> <figure class="highlight"><pre><code class="language-bash" data-lang="bash">helm <span class="nb">install </span>rf-service chart/rf-service</code></pre></figure> <p>and then setting the example config:</p> <figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="w"> </span><span class="nl">"fetcher"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ZipFetcher"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://github.com/devopsspiral/rf-service/archive/master.zip"</span><span class="p">,</span><span class="w"> </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"rf-service-master/test/resources/testcases"</span><span class="w"> </span><span class="p">}</span><span class="err">,</span><span class="w"> </span><span class="nl">"publisher"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"CaddyPublisher"</span><span class="p">,</span><span class="w"> </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"http://localhost:8090/uploads"</span><span class="w"> </span><span class="p">}</span></code></pre></figure> <p>I’m using localhost:8090 only because in this setup all the containers are in same pod and accessible via localhost on different ports. CaddyPublisher address is used by rf-srvice not external client, like browser.</p> <h2 id="conclusion">Conclusion</h2> <p>The created frontend doesn’t have styling and still needs some more work, but building it with Vue.js was real fun. After getting the whole idea of components it was quite easy to add new stuff there. Solutions to most of my initial problems could be found in great Vue.js materials or on forums. Vue.js is really great tool for creating both simple and complex apps and it is definitely a lot of learning ahead of me in this matter.</p> <p><small><a href="#toc">Back to table of contents</a></small></p> https://devopsspiral.com/articles/k8s/robotframework-service-fe/