The VISA Tool Engine (Metadata-Driven Design)

I took over a complex debug tool where the logic for different CPU cores and steppings was becoming unmanageable through traditional hard-coding. By "grokking" the code to find its "beating heart," I realized the project could be re-architected as a metadata-driven engine centered on XML.

I shifted the design so that the XML schema defined the GUI and silicon features—such as a checkbox to forward VISA signals over JTAG—allowing the tool to auto-detect and deploy functionality without any further code changes. This "Configuration as Code" approach allowed me to support years of new hardware iterations by simply updating XML definitions scraped directly from RTL.

Automated Parsing - The XSLT Experience

Part of maintaining the VISA tool involved manually stitching together different silicon code blocks with their corresponding RTL flow so the signals would be available to the whole engine. The tool would allow you to select signal sources in the silicon and spit out JTAG commands to enable a direct connection to LA pins or the on-chip LA (OCLA).

I developed an engine to directly convert RTL into XML compatible with the tool. This saved tremendous time and allowed a quick turnaround for new steppings and layouts. I also got some experience outside of linear code with XSLT.

A Simple Config File

The VISA tool was also highly dependent on the stepping and family of a processor. Each new power-on of new silicon required me to re-compile the project and update the source code. Wary of this extra maintenance step I added a config file. It was a simple two-day fix that saved me about 2 hours per month, but also enabled anyone to add a project without my express involvement.

My manager quietly wrote it down and later surprised me by praising me for forward thinking in my yearly focal. Since then I’ve learned value does not always correspond to effort, and gave me respect for simple QOL improvements.

Flashing 100 Nodes (Fleet Scale Operations)

While in my PAE role, I maintained roughly 100 Knights Landing nodes that allowed us to reproduce issues with OEMs. Stuff you’d only see once in a million happened regularly when you start doing a million things quickly and in parallel. We used pdsh regularly. You think very differently about manually flashing BIOS when it needs to be done 100 times with a tough-to-reach harness.

I helped execute and collect data for a heat experiment. We visualized nodes getting hotter at the tops of racks, experimented with the rack doors open. The results of the experiments were somewhat of a moot, the minor heat differences didn’t affect performance. Notably, cables got in the way of airflow, but the connections were mandatory. So many loose connections. Each node had its own personality, and some were very stubborn.

The Cygwin Wall (Environment Parity)

We had an old internal patch for ipmitool to enable parsing of some extra return codes bytes for automation. To set up the build environment I downloaded the source over git and followed the steps to compile in Cygwin. The build script would fail, I’d fix the failure, then it would crash somewhere else. I wasted a week just trying to get the thing to compile.

It turned out that I had downloaded using Windows git rather than Cygwin git. This was a major issue because line endings were unix style. Once I re-downloaded the source everything compiled perfectly. Always be wary of line endings. Take a step back when you run into a wall.

Reinventing TI Basic

When I was in Junior High I learned basic programming on my TI-85. I was particularly proud of a program that would comb through possible numerator/denominator combinations to convert a ‘float’ into a fraction. I had a bad backup coin battery so my file system would get wiped pretty regularly, so I got good at rewriting that converter.

Then one day I was looking through the list of commands and noticed an internal ->Frac function that had already implemented what I wanted. Two things stuck in my mind after that: (1) You get good at doing things by doing it again and again, (2) look for things you assume are not there.

The QuakeCon Hot Swap

The year was 2001, just before 9/11. I flew my PC to a LAN party in Texas. I was using a cheap IDE raid controller at the time. It had a bad eprom after a failed firmware update. Thinking quickly, I ordered another controller. Booted with the good firmware so I could get to a flash point, then hot-swapped the eprom to flash the bad. Was able to recover and even return the controller.

Reading Like a Robot (Deep Log Debugging)

When assisting a colleague with stubborn system failures, I found that standard keyword searching through logs was failing to find the root cause. I applied a systematic methodology I call "Reading like a robot," where I refuse to skip any part of a stack trace or log file, even the unrecognized or "noisy" lines.

By meticulously connecting every word and mapping the execution flow sequentially, I was able to identify a specific logic deviation hidden in the background noise that others had scrolled past. This brute-force attention to detail proved that even the most "messy" telemetry can be solved if you check your assumptions at the door and follow the machine's actual path.

RAPL & Matplotlib (Telemetry Visualization)

In early 2019 I took a machine learning course. Jupiter notebook and time series db exercises. A few python libraries stuck out at me as very useful at work: pandas and matplotlib. I had a decent amount of data collection when stressing Running average Power Limiting functionality. Reports would spit out but it was really hard to get a good picture on what was going on.

With the new tools in hand I was able to scrape current runs, even old logs, and bug reports, to provide clear graphs sweeping through request or expected vs actual power limiting results. An esoteric ‘10% below expected’ would turn into a graph that would cap out at 500W like a truncated pyramid - a clear indication that the part (or benchmark) wasn’t able to cause the system to draw that much power.

How to Grok Code

A senior dev was mentoring two of us while looking through a rather large code base. She took out some paper and wrote down how the program was structured in a kind of shorthand. By the time we were finished we had a map and a clear understanding of the flow. Since then I’ve developed my own shorthand to systematically map out unfamiliar code. I know I’m done when I can see the ‘heart’ of the code.

The Beating Heart

To follow up on code grokking, once I find the ‘beating heart’ in a system I know I’ve found the perspective of the original developer and everything becomes clear. Some of these are straightforward, a loop handling events, a main function that does the heavy work. One particularly difficult ‘heart’ turned out to be outside the code entirely, the main logic was all contained inside an XML file that described RTL pathways to the LA pins. It was a rather elegant solution but a pain to debug until I understood the process.

Expect Fragments (Serial Validation)

I wrote an automation harness that allowed me to validate TGUI applications in a OS platform agnostic way. It was rather nifty, using TCL and the expect library to handle reading and writing to the screen. Unfortunately, when I started using expect for serial output (EFI mode) things got dirty. I learned that serial output has no guarantee to preserve words—menu items would often get clipped. I mostly got around this with generous regular expressions. In theory, the flows should have worked, but in the real world, the data was noisy. However, accepting that noise as a constraint paid off. By building a robust parser that tolerated the fragments, I was able to shift the NIC validation process from 100% manual to 90% automated, saving hours of tedium and proving that even messy interfaces could be tamed.

Crazy Callbacks

I used a clever callback pattern when chunking data over PECI - sending multiple read requests and mapping the responses with tailored callback functions to de-serialize the data. This asynchronous approach, detailed in our 2022 DTTC paper "Measuring PECI Performance," allowed us to chain requests without blocking, improving throughput by nearly 100x (from ~57 to ~5300 commands/sec) and proving the software stack could finally keep up with wire speed.

Throughput Gotchas

Measuring throughput is a tricky question, especially when data is encapsulated at several levels. This is pretty standard over IP but it’s more ambiguous when counting packets over SMBus or other internal protocols. Our DTTC paper tackled this, but I was never quite satisfied to provide a standard MB/s measurement. Because the protocol overhead was so heavy, I argued that "Commands per Second" was the only honest metric, even if it wasn't the big megabyte number stakeholders wanted to see.

Curl over Postman

I prefer Curl over Postman for the same reason I prefer text over screenshots - design tests for reproducibility ease - compartmentalize into a series of batchable steps - essential for bug reports.

Venv is like Hot Glue

Venv is like hot glue - a stop gap solution that becomes standard - docker feels the same - has better utility these days.

The RAKP Security Catch (CVE-2013-4786)

While implementing the RMCP+ protocol stack from scratch in C, I performed a deep dive into the RAKP authentication spec. To properly debug the session negotiation, I had to patch the Wireshark (then Ethereal) source code to correctly parse the custom packet exchanges I was seeing.

During this process, I identified a fundamental flaw: the BMC would return a salted hash of the user’s password before the client proved their identity, essentially enabling offline dictionary attacks. Although my concerns were dismissed by senior engineers at the time, this exact vulnerability was later confirmed as industry-wide flaw CVE-2013-4786. This solidified my conviction to trust technical data over hierarchy and to maintain meticulous documentation for critical findings.

The Kernel Patch (Topology & Architecture)

We had a module flooding dmesg on custom silicon due to rigid topology assumptions. I wrote and compiled a kernel patch to add the necessary flexibility. This eliminated the noise, allowing the team to focus on real bugs. While the patch couldn't be upstreamed due to red tape, the experience validated that I could dive into kernel code to unblock the validation process.

Negative Testing (The Yield Use Case)

I had a big PECI coverage miss, it was rather embarrassing. Because we had a huge list, I was only checking what I knew existed, not what shouldn’t exist. Well, some unexpected endpoints did surface, more echoes of what they should be, but those endpoints should not have been visible. I needed to pivot away from just reading known devices to scanning and including negative space.

I recalled a conversation with an expert who had scanned through the BDF region before and adapted that to PECI space. To do that I needed a list of all possible combinations in PCI space - which fit very neatly with a generator pattern. One of the first times I really felt yield improved the readability and reliability of my code.

After I got it running I still saw ‘extra’ devices popping up, but they turned out to be actual hardware, which was easy to correlate with lspci. It was a huge space to cover, and a full run would have taken almost a week, but with the combination of the bulk transfer API and fast negative returns the expected execution time went from a few days to under 10 hours.

The Texas Power-On (Strategic Career Pivot)

I went to Texas for a power-on during a transition period where I was doing basic execution work and looking for a long-term technical fit. Since I didn’t have a strict task list, I decided to shadow the experts and jump into the immediate fires on the floor. I ended up peer programming with a group focused on high-level security features like TDX and Secure Boot.

We clicked so well that when they (also) returned to Oregon, they reached out to pull me into their organization. I wasn't just helping with a bring-up; I was recruited to seed a second team under the Security & Manageability umbrella, which launched my five-year run as the Lead Manageability Validation Automation engineer.

The Failed Microsoft Demo (Preparation & Redundancy)

I had lightweight RMCP+ library we were going to demo to MS. At the last minute, the firmware developers enabled and enforced encryption, which I had not implemented yet. These days I never trust a demo. Demos are great, but always provide a backup video and slides.

RMCP+ Socket Optimization

Our dev team was tracking down a problem where RMCP+ sessions were piling up in our validation cycle, overloading the BMC and causing instability. We had a TCL based framework, but the heavy lifting was done inside DLL files. I had been doing some digging and found the source code. When I mentioned this to the dev lead he asked me to look into the source code as a last resort, as far as they could tell their code was clean.

After a bit of sleuthing I found code responsible for opening and closing the sessions on the client side, there was a return between them, so the socket was never closed. This one fix gave our test suite the stability to add unattended validation cycles and focus on nightly test reports. It was a game changer, and the experience taught me a single line could be worth weeks of effort.

To SCM or Not to SCM (Integration Reliability)

Early in my career, I was tasked with helping our team transition from SVN to IBM ClearCase during a mandate from our international partners. It was an exercise in pure persistence; while the infrastructure was in flux, I tag-teamed with our resident SCM lead to keep our daily validation builds running. I had to resort to intense "mergify" sessions and manual elbow grease to ensure the transition didn't break the active development pipeline.

ClearCase has a unique way of mixing software recipes, and I handled several emergency integration sessions to maintain code integrity. While I’ve always enjoyed a good diff, those sessions taught me that the best work is the work you avoid through proper architecture. Since then, I’ve been extremely careful about identifying and preventing the ways teams unintentionally fork code and create technical debt.

Estimates: High vs. Low (The Code Archaeologist)

I was asked to estimate how long it would take to ramp up and deploy a project that had recently been recovered from an old backup. This was my ‘code archeologist’ time to shine. I recently took an estimation course and learned how setting high and low estimates improved accuracy, but my manager wanted real dates.

I began execution over several months, with the estimate plan driving me and my contractor while my manager was on sabbatical, we hit our milestones well, and saved the complicated unknowns for last. It was a very smooth execution. On completion and review, I asked for feedback. The only thing I got was “Maybe you could have added a range of estimates” - I don’t expect managers or people to be consistent, but it was a big trust building time when I casually reminded him I was tracking high and low estimates off the side of the spreadsheet.

The PythonSV API Thrash

I was leveraging a PythonSV framework (c-scripts) for silicon power-on to improve test coverage. My manager at the time specifically asked me to add API-level tests for visibility. I knew they were technically redundant, but I followed the direction to ensure the team had the completeness they were looking for. A year later, during a review, I was berated for that exact redundancy by the architecture team.

That "unnecessary thrash" was a huge lesson in code ownership. I learned that just "following orders" isn't enough when you're working in a complex framework. Now, if I’m asked to implement something that creates technical debt or redundancy, I make sure there is a clear "paper trail" or attribution in the code itself. I give proper credit to the source and document why the decision was made, so when the architecture is reviewed a year later, the context isn't lost.

Synergy of Validation Working with Architect and Dev

I really enjoyed working closely as validation developer with a BMC developer and Architect - We met daily in an informal SCRUM like context. Architect would describe the focus, and we would push back on feasibility and worth. We got a lot of performance out of PECI, creating a bulk transfer API with hooks to prove we were moving at line speed - Restful API extensions to directly compare LAN overhead with PECI ‘wire’ time (DMA was considerably faster than wire) - this resulted in a DTTC paper on measuring the performance of PECI.

Always Fix the First Error

I stumbled upon a ‘gotcha’ moment very early into learning how to code in school. Everyone gets their first missing semicolon sometime. When compilation errors started piling up, I naively and dutifully started fixing every error before hitting ‘compile’ again. It was infuriating and stressful. I had no idea what was going on, why were there so many errors. That is, until I noticed compilation errors had a waterfall effect. One missing semicolon can mess up the interpretation of the reset of the file. The opposite of reading a stack trace: always fix the first compilation error.

Count the Ok's

Whether it’s deciding what restaurant to go to or voting on the best way to execute a plan, voting isn’t the best option, and rather counterintuitively it isn’t the fairest either. Important decisions should be made carefully, and allowing people to weigh in on all the options that are ‘okay’ drastically improves the results.

The Conflict Management Lesson

There have been several times in my career where I’ve been convinced I was right but wasn't in a position to change the direction. I’ve learned to disagree and commit, and in those cases, the results usually went well because the team stayed aligned.

In fact, I’ve often "embraced the suck" and intentionally became a subject matter expert on the critical things others didn't want to touch—like ClearCase, DTAF, or the complexities of XML and RTL scraping. By mastering those "unpopular" parts of the stack, I was able to bridge the gaps and keep the validation pipeline moving. That said, I still maintain technical boundaries for the sake of the product; for example, I’ll advocate strongly against putting test-level code like Pytest into production logic to keep the architecture clean.

General Thoughts & Philosophy

• The "Class 1" Assumption: It’s a dangerous assumption to code for an end user—modal dialogue boxes, frameworks—assuming code will remain on top. Frameworks should be viewed as debuggers for internal libraries.
• Content is King: Automation is Queen but Content is King. This mantra focused our team. Don't let frameworks overshadow actually getting things done.
• Can AI Really Think? Question needs to shift from 'can it think?' to ‘Can thoughts live on paper?’ Yes. Language is the single most important invention created by humanity.
• 2 and 7 Year Cycles: Roughly every 7 years I reinvent myself. Projects tend to finish by two years.
• Chopstick Coding: The concept of developing at arms length (i.e. through a serial port, over SSH etc). Doing so is difficult, like typing on a keyboard with chopsticks.
• Meetings: Inversely proportional to how many people attend.
• Communication: Don't reply hot, or on mobile, voice is best.