Remote Serial Port Access

Not just because of COVID-19 people may need to work from home, there is a lot of case that people need to access a serial port on a remote setup, such as access a Raspberry-Pie UART on a host PC. VirtualHere provides a commercial solution to create a Virtual USB on local host to virtually represent the USB port on the remote host. This works pretty well with FTDI USB-to-Serial chips. There are many other solutions. Specially if you like me want to have a simple manageable solution, such as using script to do the job. A good example I found from here: http://fabacademy.org/archives/2015/doc/WebSocketConsole.html, which create a web view of UART output with purely Python script.

Too lazy to check what the license is for posting the code. So I won't try to copy and paste the article here. It was published in 2015, a little old, but I assume it should still work well. When I tried it out on Windows recently, I do see some issue, mainly due to known issue for python package on Windows.

Python 2.x is too old. So the first thing is updating for Python 3.x, such as convert all "print '...' " to "print('...')". There is also some tab/space mixing issue need to be fixed.

For using pyserial to open serial port on windows such as COM6, no need to use the awkward device name "//./COM6", just open "COM6" should be fine.

The big issue is the multiprocessing pickling error on Windows, will get error:

self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

Google it will get a bunch of similar topic, such as this pytorch one: use if __name__ == '__main__': to protect your main function (more details here). Can also try to run the script in command prompt. If the error persists, then you can try to set num_worker of DataLoader to zero.

And with Stackoverflow. As explained here: Windows doesn't have fork, so there's no way to make a new process just like the existing one. So the child process has to run your code again, but now you need a way to distinguish between the parent process and the child process, and __main__ is it.  

'if __name__ == '__main__'' is already in the original code, the num_worker seems is pytorch specific. So for this case, looks like need to fix empty input problem, maybe need to force the serial port read as a blocking read? Tried changing serialworker.py as:

self.sp = serial.Serial(SERIAL_PORT, SERIAL_BAUDRATE, timeout=0)

but it doesn't work out. So maybe need to consider using pyserial in thread instead.

--- to be continue ---

Though I wasn't able to figure out the reason, but I'm able to get it working by removing the serialworker class completely, put all serial I/O code to the worker of multiprocessing.Process.

Now new issue surfaced: 

1) appdata\local\programs\python\python38\lib\site-packages\tornado\platform\asyncio.py", line 79, in add_handler

    self.asyncio_loop.add_reader(
  File "c:\users\kokat\appdata\local\programs\python\python38\lib\asyncio\events.py", line 498, in add_reader
    raise NotImplementedError

This is a known issue as tornado issue 2068, and also mentioned in Tornado site. The fix/workaround is adding below in \Lib\site-packages\tornado\platform\asyncio.py

import sys

if sys.platform == 'win32':
     asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

Or add above line in main, right after the if __name__ == '__main__' check (add import asyncio)

2) error TypeError: __init__() got an unexpected keyword argument 'io_loop'

One solution is as User a local Jupyter Notebook, downgrade tornado with:

 pip install tornado==4.5.3

This may also get rid of the first issue. But I more prefer to use the new version of Tornado. So the other solution is remote the obsoleted keyword 'io_loop'.

3) checkQueue() is invoked by a scheduler, with 100 as interval, the 'if' check in the code only can pick one line serial port output from the queue, which will lead to bad throughput. Should change the 'if' check as 'while', like this:

    while not output_queue.empty():

        message = output_queue.get()

4) WebSocketHandler is not working as expected, one reason is that main.js failed to be loaded. Other reason could be with the Javascript and the html script.

--- to be continue ---

After some research, debugging and testing, I got a solution to have a html file with inline JavaScript. It's not perfect, still has some minor issue, such as the 'clear' button doesn't work as expected. The code is like below (modified based on tutorialspoint code):

<!DOCTYPE html>

<html>

   <head>

      <meta charset = utf-8>

      <title>WebSocketConsole</title>

      <body>

         <section id = "wrapper">

            <header><h1>WebSocketConsole</h1></header>

            <style>

               body { margin: 0px; padding: 20px; }

               #status { padding: 5px; color: #fff; background: #ccc;}

               #status.fail { background: #c00; }

               #status.success { background: #0c0; }

               #status.offline { background: #c00; }

               #status.online { background: #0c0; }

               #received { width: 500px; height: 400px; border: 1px solid #dedede; font-family: "Lucida Console", Courier, monospace; font-size: 10px;

                           overflow-y:scroll; display:flex; flex-direction:column-reverse}

               #cmd_str { width: 97%; }

               .message { font-weight: bold; }

               .message:before { content: ' '; color: #bbb; font-size: 14px; }

            </style>

            <article>

               <p id = "status">Not connected</p>

               <p>Users connected: <span id = "connected">0</span></p>

               <p>Data received from serial port</p>

               <div id="received"></div>

               <button id="clear">Clear</button>

               <p>Send data to serial port</p>

               <form onsubmit = "send_cmd(); return false;">

                    <input type = "text" id = "cmd_str" placeholder = "type cmd and press enter" />

               </form>

            </article>

            <script>

               var state = document.getElementById("status");

               var connected = document.getElementById("connected");

               var received = document.getElementById("received");

               var clear = document.getElementById("clear");

               clear.onclick = function() {

                   received.empty();  //TODO: this does not clear the log. Need fix

               }

               var cmd_str = document.getElementById("cmd_str");

               if (window.WebSocket === undefined) {

                  state.innerHTML = "sockets not supported";

                  state.className = "fail";

               }else {

                  if (typeof String.prototype.startsWith != "function") {

                     String.prototype.startsWith = function (str) {

                        return this.indexOf(str) == 0;

                     };

                  }

                  window.addEventListener("load", onLoad, false);

               }

               function onLoad() {

                  websocket = new WebSocket("ws://"+location.host+"/ws");

                  websocket.onopen = function(evt) { onOpen(evt) };

                  websocket.onclose = function(evt) { onClose(evt) };

                  websocket.onmessage = function(evt) { onMessage(evt) };

                  websocket.onerror = function(evt) { onError(evt) };

               }

               function onOpen(evt) {

                  state.className = "success";

                  state.innerHTML = "Connected to server";

               }

               function onClose(evt) {

                  state.className = "fail";

                  state.innerHTML = "Not connected";

                  connected.innerHTML = "0";

               }

               function onMessage(evt) {

                  var message = evt.data;

                  if (message.startsWith("connected:")) {

                     message = message.slice("connected:".length);

                     connected.innerHTML = message;

                  } else {

                    //console.log("receiving: " + message);

                    received.append(message);

                    var linebreak = document.createElement("br"); //refer to link

                    received.appendChild(linebreak);

                  }

               }

               function onError(evt) {

                  state.className = "fail";

                  state.innerHTML = "Communication error";

               }

               function send_cmd() {

                  var message = cmd_str.value;

                  cmd_str.value = "";

                  websocket.send(message);

               }

            </script>

         </section>

      </body>

   </head>  

</html>

Get Fun with Google Assistant

I first time read about it from Inquirer post of a Mod of Intercom device. And here is the Youtube clip: l
So I started to read about Google Assistant SDK. So basically there are two choices, based on what hardware platform you have: 1) with a Python-based library; 2) use gRPC API.
The Python lib option is a turnkey solution for anyone who wants to quickly integrate the Assistant into a prototype device. It supports linux-armv7l(include Raspberry Pi3) and linux-x86_64. gRPC is a modern open source high performance RPC framework that can run in any environment. It was initially created by Google, and was turned to open source in 2015.

For the Python solution, refer instruction here. If run into problem with 'python3 -m venv env', make sure the 'env' folder does not exist, also, refer to this stackoverflow post. If you already had done this step before, and want to upgrade, rename the old folder in case you have config/json file or code there, run the venv command, then copy over config file and your code after that.

学习舵轮(kubernetes)和埠(docker) - kind

Minikube and Kind are the two recommended by the official Kubernetes website.

Refer this for the comparison between minikube, kind and k3s. Here is a brief description of Kind:

Kind is another Kubernetes SIGs project but is quite different compared to minikube. As the name suggests it moves the cluster into Docker containers. This leads to a significantly faster startup speed compared to spawning VM.

For install, refer to https://kubernetes.io/docs/setup/learning-environment/kind/
On Linux, the simple way is to run below to download the single executable:

curl -Lo ./kind https://github.com/kubernetes-sigs/kind/releases/download/v0.7.0/kind-$(uname)-amd64

Change it to be executable and add it to path, all done. If like to learn about the hot 'Go', you can use Go to get it, and even build from source.
It has dependency of docker. To install docker: refer to this old post.
Or on Ubuntu, run: sudo apt install docker.io

To play with kind, need to install kubectl:

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

As kind Quick Start mentioned, after kind and kubectl installed, things can be tried are:

• Creating a Kubernetes cluster is as simple as kind create cluster

• List your kind clusters: kind get clusters

• Interact with the created default cluster: kubectl cluster-info --context kind-kind

• Delete cluster: kind delete cluster 
  

• Load image to kind-2 cluster: kind load docker-image my-custom-image --name kind-2

When I  try to use my docker image (by running 'docker pull quyq/ifun:v0.3') to create cluster, I got error:

ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged kind-control-plane cat /kind/version" failed with error: exit status 1


This sounds like the same issue as kind issue 1288.  Run kind create cluster --image d7fe9f82ee94 -v 1 

shows: cat: can't open '/kind/version': No such file or directory

Per BenTheElder's comment, snap is in the known-issue document, the snap docker package

has a number of issues, e.g. no access to temp directories. I don't recommend snap for docker

and we don't really support this. Resolution was to install docker via apt-get detailed on their

page here. But for me I was usng the curl command installed kind, and even upgrade to version

0.8.1 doesn't help. I have no issue to create the cluster with default kindest/node image.  By

 checking this, turns out I need a customized docker image for kind.

Windows10 运行Ubuntu - WSL2

参见2019年2月的blog：Running Linux on Windows。WSL2是微软第二代Windows Subsystem for Linux。WSL2与WSL的区别可以参考Comparing WSL 2 and WSL 1。简单说，WSL1提供了Linux shell支持，但底层是把Linux移植到Windows内核上，所以不支持系统调用(system call)，不支持Linux Kernel，有诸多限制，不易扩展，不易移植。WSL2则是把Linux内核运行在虚拟机上，支持完整的system call，Linux程序可以直接拿过来编译运行，缺点主要是需要打开虚拟机支持（Hyper-V），与现行的VMWare和Virtual Box有冲突(好消息是VMWare和Virtual Box的新版本都将支持Hyper-V)；另一个问题是固有的Linux对Windows文件系统的支持问题(虽然Linux支持NTFS和FAT，可由于Windows文件系统没有Linux文件权限支持（如root, group, owner),也不支持Linux symbolic link，在cross OS的文件系统方面有一定限制。
Comparing WSL 2 and WSL 1提到只有如下两种情况可以考虑使用WSL1而不是WSL2:

• Your project files must be stored in the Windows file system.
  • If you will be using your WSL Linux distribution to access project files on the Windows file system, and these files cannot be stored on the Linux file system, you will achieve faster performance across the OS files systems by using WSL 1.
• A project which requires cross-compilation using both Windows and Linux tools on the same files.
  • File performance across the Windows and Linux operating systems is faster in WSL 1 than WSL 2, so if you are using Windows applications to access Linux files, you will currently achieve faster performance with WSL 1.

In fact, there is other cases you may want to use WSL1, such as WSL2 ticket #4322 for accessing Windows USB device from WSL: Ubuntu 18.04 Serial devices are no longer mounted properly. It’s clearly called out in the FAQ that there is currently no support for USB in WSL2. It is a highly desired feature, but unfortunately still not available after almost two years as of today. Good news is: you can switch back to WSL1, and it is very easy to do the 'downgrade', no installation, no reboot needed, just run one command from Power Shell:

wsl --set-version Ubuntu 1

The same Linux Distribution installed for WSL2 should still work as is. 

WSL support很早就在Windows Insider build里提供了，并且正式出现在updated to version 2004, Build 19041 or higher。参见微软link安装指南。微软的安装指南里没有提到直接安装WSL2，而是谈到如何从WSL1升级到WSL2, 实际上用户不需要安装WSL1, 然后再升级到WSL2。安装WSL2可以直接从安装Linux Kernel package 开始。然后去微软AppStore选个Linux安装。WSL2本身只提供在Windows上运行Linux的支持，真正的Linux系统需要另外安装，用户可以选择Ubuntu, Debian, Alpine等。不同发行的Linux都基于相同的Linux内核（可能是不同内核版本），可能不同的内核配置，不同的软件包，图形界面等。详情参见Wikipedia。
安装完Linux，运行wsl用户会发现只有terminal终端界面。如果想要图形GUI界面，则需要借助x server的支持，详情参见Ubuntu Wiki。 

Update@Jul 28, 2022:

If you are on be on Windows 11 Build 22000 or later, you can take the latest feature from WSL which supports Linux GUI APPs. Please refer to https://docs.microsoft.com/en-us/windows/wsl/tutorials/gui-apps. If you are still on older Windows 10 like me, you may refer to this WSL issue 4106 on github for some reference information. For me, Cygwin-X server is not working for WSL2, though it works for my Cygwin GUI APPs, probably due to lacking the capability to disable access control.VcXsrv Windows X Server download | SourceForge.net works for me, with access control disabled. Also works with MobaXterm (free Home Edition) built-in X server in similar way.And one more note on this is: for setting the DISPLAY environment variable on WSL2 side, I have to use the my wireless IP which is actively connected, and my Ethernet cable wasn't connected. Cannot use the command from above Ubuntu Wiki link as:

export DISPLAY=$(awk '/nameserver / {print $2; exit}' /etc/resolv.conf 2>/dev/null):0 # in WSL 2

The output of above command is IP for Ethernet adapter vEthernet (WSL). For some reason it just doesn't work, not sure is it due to company's network setting, or some other reason. If have VPN connected, may need to use the VPN IP. So if it is not working, try all the IP you can get from ipconfig's output, except the vEthernet of WSL. Loop back IP 127.0.0.1 won't work either.

 

If seeing error like "cannot execute binary file: Exec format error", refer to this Stackoverflow post: https://stackoverflow.com/questions/42120938/exec-format-error-32-bit-executable-windows-subsystem-for-linux. Likely user installed 64bit Linux and was trying an 32bit executable. Need to enable 32 bit support with command:

sudo dpkg --add-architecture i386
sudo apt-get update

If seeing the error with GCC, might also need to do:

sudo apt install gcc:i386

and maybe below as well:

sudo apt-get install gcc-multilib g++-multilib libc6:i386

And likely this only works with WSL2. If you updated from WSL1, running

 wsl --set-default-version <Version#>

won't affect distribution already installed on your PC. You can run: "wsl -l -v" to check distribution name and which WSL version it is. To change WSL version, run:

wsl --set-version <distribution_name> 2

and it will take a little time to convert.

Refer to bash - What is the home directory on Windows Subsystem for Linux? - Super User for how to access the guest distro folder from host Windows OS (\\wsl$\Ubuntu). From guest distro such as Ubuntu to host Windows C: drive, it is mounted as: /mnt/c  

 

Update on Nov 24: per several website such as xda-developers.com, the latest WSL also enabled Windows 10 user to run GUI application. It also brings in systemd support.

For Audio support, WSL on Win11 should have it enabled already. If on Win10, may follow instruction from https://x410.dev/cookbook/wsl/enabling-sound-in-wsl-ubuntu-let-it-sing/ which has clear step by step instruction and does not open unnecessary permission for utilizing PulseAudio

 

Tcl parsing xml test description file

#This script is using xml to define testcases and testsuite. With this, it is very easier to create new testcases, without need to update the script framework;
#Testunit would be the basic unit in the xml, which corresponding to a proc in tcl script, as the function block, such as make connection, iperf traffic test, set/get MIB etc;
#A testcases.xml is a predefined testcase set, which only need to be updated when a new test scenario is needed;
#Each testcase contains one or more testunit, plus setting and expect result as attr;
#Testsuite xml is user defined collection of testcases and testunits;

#Each xml must have one and only one root tag, and struct like this:
# <root_tag attr1=xxx attr2=yyy ...>
#     text
#     <sub_tag1 sub_tag1_attr1 ...>
#     <sub_tag2 sub_tag2_attr1 ...>

#In topologic, each tag is a node, attr is a list, and sub tags are sub nodes;

#http://wiki.tcl.tk/1740 convert content to xml format, default tag is 'document'
proc asXML {content {tag document}} {
    set XML_MAP {
        < &lt;
        > &gt;
        & &amp;
        \" &quot;
        ' &apos;
    }
    return <$tag>[string map $XML_MAP $content]</$tag>
}

#http://wiki.tcl.tk/3919 A little XML parser
#convert xml to: tag {list: attr} {list: #text {_text_}, subTag1, subTag2 ...}
proc xml2list xml {
    # trim only deal with the begin and end of the whole xml
    #puts "dbg trim: \[[string trim $xml " \n\t<>"]\]"
    regsub -all {>\s*<} [string trim $xml " \n\t<>"] "\} \{" xml
    # above regsub replace '>' and '<' with brace if no text between them. It also remove \r \n between '>' and '<'
    #puts "dbg after regsub: \[$xml\]"
    set xml [string map {> "\} \{#text \{" < "\}\} \{"}  $xml]
    #puts "dbg after map: \[$xml\]"
   
    set res ""   ;# string to collect the result  
    set stack {} ;# track open tags
    set rest {}
    foreach item "{$xml}" {
        #puts "dbg item => $item"
        switch -regexp -- $item {
            ^# {append res "{$item} " ; #text item, or "{#text [lrange $item 1 end]} ", origin post not work as it said: "{[lrange $item 0 end]} "}
            ^/ {
                regexp {/(.+)} $item -> tagname ;# end tag
                set expected [lindex $stack end]
                if {$tagname!=$expected} {error "$item != $expected"}
                set stack [lrange $stack 0 end-1]
                append res "\}\} "
            }
            /$ { # singleton - start and end in one <> group
                regexp {([^ ]+)( (.+))?/$} $item -> tagname - rest
               #set rest [lrange [string map {= " "} $rest] 0 end] <= not work if attribute like: attr1="var=x"
                regsub -all {=\s*"} $rest " \"" rest
                regsub -all {=\s*'} $rest " '" rest
                set rest [lrange $rest 0 end]
               #with above two lines, attribute has to be single/double quoted
                append res "{$tagname [list $rest] {}} "
            }
            default {
                set tagname [lindex $item 0] ;# start tag
               #set rest [lrange [string map {= " "} $item] 1 end] <= same as above, not work if attribute contains equal
                regsub -all {=\s*"} $item " \"" item
                regsub -all {=\s*'} $item " '" item
                set rest [lrange $item 1 end]
               #with above two lines, attribute value has to be single/double quoted, but the value should not contain such as ="
                lappend stack $tagname
                append res "\{$tagname [list $rest] \{"
            }
        }
        if {[llength $rest]%2} {error "att's not paired: $rest"}
    }
    if [llength $stack] {error "unresolved: $stack"}
    string map {"\} \}" "\}\}"} [lindex $res 0]
}

proc xputs {msg {log 0}} {
    variable xputs_level
    set indent ""
    if {[info exists xputs_level] && [regexp {^\d+$} $xputs_level]} {set indent [string repeat {  } $xputs_level]}
    if {$log ne "0"} {send_user "$indent$msg\n"
    } else {puts "$indent$msg"}
}

proc print_tree {lxml level} {
    #puts "level = $level"
    puts "[string repeat {  } $level]tag = [lindex $lxml 0]"
    puts "[string repeat {  } $level]att = [lindex $lxml 1]"
    incr level
    foreach sitem [lindex $lxml 2] {
        print_tree $sitem $level
    }
}

#testcases.xml is xml file contains pre-defined testcases. This xml file would be processed to generate an array of testcases definition, no execution here
#testcase name is the index of the array, such as 'TC-SCAN-3-1', and each element of the array defines how to run the testcase, include setting, expect result, and testunit needs to be executed
#the analyze result will be set in variable testcases which is an array;
#Each testcase must have at least one TU, and setting/exp_res passed to the TU/TUs as arguments
#The test xml such as a Test Suite may have one or more testcases. All testcases within one xml node shares all settings, such as repeat counter, timeout setting
#Within one xml node, may have multiple setting, the later setting will be appended, override old value, but no reset. Setting won't be passed to sibling nodes
#Setting would be passed to direct child as reference, where a setting/expect node will update them, and a TC/TU node will take it as input args
#TODO: Currently setting won't be passed cross a node. Setting inherit more than parent? Setting per testcase? Good side: avoid pollution. Bad: cannot share       

proc process_testcases_xml {tc_xml_file} {
    variable testcases
    variable tc_subs    ;#for subs defined in testcases_xml
    set fp [open $tc_xml_file]
    set testcases_xml [read $fp]
    close $fp
   
    #puts "Original xml:\n$testcases_xml"
    set l_tcs_xml [xml2list [regsub -all {(^|\n)\s*#[^\n]*\n} $testcases_xml {\1}]] ;#remove comment and covert xml to list
    set tc_sub_list {}
    #puts "tcs xml list: $l_tcs_xml"
    #print_tree $l_tcs_xml 0
    set ltcs [lindex $l_tcs_xml 2]
    foreach tc $ltcs {
        #puts "=>$tc"
        #=>testcase {name TC-SCAN-3-1} {{setting {Operation Link, Chan 1, Mask 1, Repeat 4} {}} {expect {result success} {}} {run {name TU_scan_test} {}}}
        if {[lindex $tc 0] eq "sub"} {
            set tc_subs([lindex [lindex $tc 1] 1]) $tc
        } else {set testcases([lindex [lindex $tc 1] 1]) [lindex $tc 2]}
    }
    #puts "\n _dbg_: dump tc_subs:" ; foreach {key value} [array get tc_subs] {puts "  $key => $value"}
}

proc get_sub_xlst {lxml sub_name} {
    set found false
    if {[lindex $lxml 0] eq "sub"} {
        array set attrs [lindex $lxml 1]
        if {[info exists attrs(name)] && $attrs(name) eq $sub_name} {return $lxml}
    }
    foreach sitem [lindex $lxml 2] {
        set found [get_sub_xlst $sitem $sub_name]
        if {$found ne "false"} {return $found}
    }
    return $found
}

#xlst is a three elments list of a xml node: node_tag, list of attrs, list of sub_node; param level is for debug purpose
#p_setting and p_exp_res is parent setting and exp_res, for current node and current node's sibling nodes
#No inherit of setting and exp_res as that would be messy, error prone and complex. So here allow to pass optional setting as attr of TC.
#Note: there is two ways to run a TC testcase, either use 'run TC-xxx' to start a predefined TC, or have a tag's name started with "TC-"
#Add global AbortTC to abort all testcases whenever needed. Not abort TU as user may still want to do TU_TEST to dump data
set AbortTC no
proc process_xmlnode {xlst level {p_setting ""} {p_exp_res ""}} {
    variable testcases
    variable tc_subs
    variable xputs_level
    variable full_tree  ;#for processing 'use'
       
    set node  [lindex $xlst 0]
    set attrs [lindex $xlst 1]
    #index 1 in xlst is key-value paired list and 1st pair should be a testunit or testcase name like TU-SETUP-HOSTS or TC-TRAFFIC-10
    #other attr could be optional setting of the testcase, or override the TC setting, may also set like expected result
    #convert attrs as array as it is key-value pairs
    set subnodes [lindex $xlst 2]
    set xputs_level $level
    upvar $p_setting cur_setting $p_exp_res cur_exp_res
    #setting and exp_res for sub nodes
    set s_setting ""
    set s_exp_res ""
   
    #inherit parent setting/exp_res: 1st, it doesn't work; 2nd, it is a bad approach as explained above.
    #set s_setting $p_setting
    #set s_exp_res $p_exp_res
   
    #xputs "Level $level node is <$node> with attr list: ($attrs)"
    #TODO: decide what to do with the $attrs
   
    if {$node eq "setting"} {
        #put setting in upvar variable as it is for parent node, and only need to perform setting in one shot(TU-SETUP-HOSTS?) to make it more effient
        set cur_setting [dict merge $cur_setting $attrs]
        if {$::LOG_LEVEL>2} {xputs " A setting: $attrs"}
    } elseif {$node eq "expect"} {
        #put expect to an upvar variable as it is for parent node and it will be checked in testunit of the parent node
        set cur_exp_res [dict merge $cur_exp_res $attrs]
        if {$::LOG_LEVEL>2} {xputs " An expect: $attrs"}
    } elseif {$node eq "exec"} {
        #print a message, eval tcl; be careful as support is limited, due to nature of xml and tcl, as quote, slash and many special character will cause issue
        #Note $attrs is treated as dict, so duplicate key in $attrs will get lost
        #Also, for eval, set variable without uplevel will get lost as this is a recursive call of process_xmlnode
        #For exec an external/system command, too many hassle: not sure the cwd, nowhere capture the output; Use eval if needed like this: <exec eval="puts [exec date]" />
        if {$::LOG_LEVEL>2} {xputs " An exec request: $attrs"}
        dict for {op args} $attrs {
            switch -exact -- $op {
               "echo" {puts $args}
               "eval" {eval $args}
               default {puts stderr "Unsupported operation: $op"}
            }
        }
    } elseif {$node eq "run"} {
        array set run_setting $attrs
        if {[info exists run_setting(name)]} {
            if {[regexp {^TC} $run_setting(name)]} {
                variable tc_results
                variable cur_tc_result
                variable tc_idx
                variable TC_name
                variable f_csv
                               
                set TC_name $run_setting(name)
                set cur_tc_result ""
                #setting for testcase are ignored for now. Should be repeat number, timeout etc.
                #upvar $setting cur_setting
                puts ""
                xputs " <Run testcase $TC_name" 1
                if {[info exists f_csv]} {puts $f_csv "Run testcase $TC_name"}
                if {$::AbortTC} {
                    set cur_tc_result "Aborted"
                    xputs "  $cur_tc_result" 1
                } else {
                    #pick the optional setting from TC attr
                    set s_setting [dict remove $attrs name]
                    foreach subnode $testcases($TC_name) {process_xmlnode $subnode [expr $level+1] s_setting s_exp_res}
                }
                dict set tc_results $TC_name $cur_tc_result
                incr tc_idx
            } elseif {[regexp {^TU} $run_setting(name)]} {
                if {$::LOG_LEVEL>1} {xputs " <Run testunit $run_setting(name)" 1}
                if {[catch {_$run_setting(name) $cur_setting $cur_exp_res} ret_code]} {
                    send_user "\n\n**** Exception while running $run_setting(name) ****\n$ret_code\n**** Full error info:\n$::errorInfo\n****\n"
                    #log_stacktrace ;# too many as this is inside recursive call
                }
            }
        } else {xputs "\t## no-name run with attrs: [array get attrs]"}
    } elseif {$node eq "use"} { ;#no attrs, neither sub-nodes expected with use
        array set use_setting $attrs
        if {[info exists use_setting(name)]} {
            set sub_name $use_setting(name)
            set use_xlst [get_sub_xlst $full_tree $sub_name]  ;#search instead of create sub list on fly, so the sub can be declared after the place invoke it
            if {$use_xlst eq "false" && [info exists tc_subs($sub_name)]} {set use_xlst $tc_subs($sub_name)}
            if {$use_xlst eq "false"} {
                xputs "\t## Not found $sub_name for use. Ignored!"
                puts "\n _dbg_: existing tc_subs:" ; foreach {key value} [array get tc_subs] {puts "  $key => $value"}
            } else {
                #puts "*process use: $use_xlst, s=$cur_setting, e=$cur_exp_res"
                #process_xmlnode $use_xlst $level cur_setting cur_exp_res
                #used node is a container, will ignore the attrs, and directly process subnodes of it
                foreach subnode [lindex $use_xlst 2] {process_xmlnode $subnode $level cur_setting cur_exp_res}
            }
        } else {xputs "\t## Missing tag for use, useless attrs: [array get attrs]"}
    } elseif {$node eq "sub"} { ;#sub block which will be invoked by others. so no processing here. Will search the list when needed
    } else {
        variable TC_name
        variable tc_results
        variable cur_tc_result
        variable tc_idx
        set is_TC no
        if {[regexp -- {(TC-\S+)} $node -> TC_name]} {
            send_user "\n\n<$TC_name>\n"
            set is_TC yes
            set cur_tc_result ""
        }
        #only node other than 'setting', 'expect' and 'run' may have sub-nodes
        foreach subnode $subnodes {process_xmlnode $subnode [expr $level+1] s_setting s_exp_res}
        if {$is_TC} {
            dict set tc_results $TC_name $cur_tc_result
            incr tc_idx
        }   
    }
}

proc process_test_xml {xml_file {tc_xml_file ""}} {
    variable testcases
    variable full_tree
    variable f_csv

    if {[info exists f_csv]} {set create_local_mon_log no
    } else {
        set create_local_mon_log yes
        set f_csv [open monitor.log w]
    }
    #first, process predefined testcases in $tc_xml_file
    if {$tc_xml_file eq ""} {
        if {[regexp {([^@]+)@([^@]+)} $xml_file -> xml_file tc_xml_file]==0} {set tc_xml_file $::XML_ROOT_DIR/testcases.xml}
    }
    process_testcases_xml $tc_xml_file
    #foreach {tc_name tc_setting} [array get testcases] {puts "$tc_name => $tc_setting"}
   
    #start processing xml_file
    set fp [open $xml_file]
    set test_xml [read $fp]
    close $fp
   
    #below regsub is for filtering out comment line in XML which is start with '#' as the 1st non-space character of a line, which is not conforming to the XML standard. May be removed in future
    regsub -all {(^|\n)\s*#[^\n]*\n} $test_xml {\1} test_xml
   
    set full_tree [xml2list $test_xml]
    #print_tree $full_tree 0
    variable tc_results
    variable tc_idx 0
    process_xmlnode $full_tree 0
   
    if {$create_local_mon_log} {close $f_csv}
}