Chris Cormack – Tech Stuff – Page 3 – Insights from a career in information technology

PrivateLink – It’s a Kind of Magic

AWS PrivateLink is an interesting way to create an endpoint by which you can provide services to other AWS accounts. You can do this without the need to run requests through the Internet and without peering or otherwise “connecting” VPCs. What is this particular type of AWS magic, I hear you say?

Says the Amazon web site, “AWS PrivateLink provides private connectivity between VPCs and services hosted on AWS or on-premises, securely on the Amazon network. By providing a private endpoint to access your services, AWS PrivateLink ensures your traffic is not exposed to the public internet. AWS PrivateLink makes it easy to connect services across different accounts and VPCs to significantly simplify your network architecture.”

So this is a particularly interesting AWS magic trick in that we can provide services to other consumer VPCs, through the Amazon backbone. We simply do two things to make his happen:

Create an Endpoint Service, in our serivce provider VPC
Create an Interface Endpoint (linked to our Endpoint Service), in our service consumer VPC

Where this gets really interesting is that it avoids all the unruliness of network address spaces and having to deal with Network Address Tranlsation (NAT). Routing works through a network interface to the endpoint service and you don’t have to worry about the network addresses. And if the endpoint service is unavailable in one Availability Zone, well that’s not a problem because your endpoint service will load balance across multiple Availability Zones.

Not to put too finer point on it, but to get the engineering and provisioning underlying all that without lifting a finger? That’s a kind of magic.

Publishing a .NET Core App

In my last blog, I mentioned a basic Web App that I had put together using Visual Studio 2015 tooling for .NET Core. Over the last couple of days, I’ve been looking at publishing the App, and the steps involved.

.NET Core is a new beast with a lot of potential…. the aim is it will run anywhere, on anything. So to keep my costs down, I’m going to trial it on my Amazon Linux VM. Note to self, SQL Server for Linux is about 1 year down the track – unfortunate, as the MVC scaffold uses SQL Server LocalDB – so I’ll have to figure out what database I can use.

But, first things first… how to publish my App to Linux?

First step – publish the App using dotnet publish. I found that Bower was not referenced in my Path environment variable. Bower was installed with Visual Studio 2015 Professional – in a sub-folder of my Visual Studio 2015 folder. I’m not sure if it was installed because I had installed tooling for .NET Core, or if it comes by default. Anyway, once that was sorted, dotnet publish worked fine and it created a portable for me to use.

I copied all of the files in the folder that dotnet publish created over to my Amazon Linux server. (I used WinSCP for this). Then I found I needed to install .NET Core on Amazon Linux. Installation was easy but when I tried to run the dotnet CLI, I received an error. Running dotnet –info from a bash shell I saw,

dotnet: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18′ not found (required by dotnet)

After several hours of searching, I found this was due to the libstdc++ library version on my Amazon Linux distribution, which was libstdc++47. I had ensured my VM was up to date, so it seems that the libstc++ version was lagging (for whatever reason). After running the below, I was able to successfully run the dotnet CLI, with a valid response from dotnet –info.

sudo yum install libstdc++48

So, I had installed .NET core and fixed up the reference library it needed. After that, I needed a way to access my Web App. Kestrel is a web server built into .NET Core and can listen on any port you tell it to. However, it is better to use a web server as a proxy/reverse-proxy to relay request/response to Kestrel.

I already had a web server on my Amazon Linux VM. So I configured it with proxy and reverse-proxy mappings (to the Kestrel server in my .NET Core App). The calls to my web server forward on to the Kestrel server in the App, and vice-versa. I ran the App with dotnet run and checked access via the web server. All good 🙂

Almost there.

Finally, I installed supervisor to manage start/stop/restart of my App. And set up a script to ensure supervisor is re-started whenever my Amazon Linux machine restarts.

Note, this would have taken days if not for the early groundwork of several people who blog their efforts.

Now I have a Web App but no database. So next step is to get the database up and running.

Scrapy Spiders, Python Processing & Web APIs

Over the past couple of weeks, I’ve spent some time drafting a Web App for Touch Footy results. The App is built on .NET Core, and this gave me a great opportunity to review the new Visual Studio .NET Core tooling. But once I had my bare bones App, I needed some data to play with. Enter Scrapy and Python…

What I wanted was a data set that I could use in the Touch Footy App. I had a good data source and I figured my best bet was a web scraper. Scrapy made it easy for me to scrape together my test data set. It’s built using Python (hence you need some understanding of Python to use it). Python is an interpreted language. It’s great for list processing and it’s easy to read/write.

Scrapy is an open source framework for writing web crawlers, or spiders. It gives you control over how and when you execute the spiders you’ve written. And a great shell as a part of the framework to test/debug commands. After looking at other web scraping options, I decided on Scrapy as a neat way to get my data.

After a few hours coding, I had a crawler that collected the data I wanted, i.e. groups, teams, fixtures and results for my Web App. I wanted to store the data in JSON – for easy processing – and Scrapy made that easy too. It was simple then to write some Python code to process the JSON for groups, teams, fixtures and results.

All good so far, and fun to boot. My next step – how to get the data to the Touch Footy Web App? Well, Visual Studio 2015 tooling for .NET Core makes it easy to add a Web API to an MVC Web Application…. several hours later I had a working spider populating data into my App.

Spinning ever faster

The world’s not spinning any faster… but working in IT these days sees my thoughts spinning faster than ever. I’m a career IT guy and I have to stay on top of this stuff. Not just for my career, but it’s just who I am. It’s in my DNA.

I’ve done this tech stuff for a long time, and I’m pretty quick on the uptake. But it’s getting crazy out there. The rate of tech advance increases day by day, creating new or deeper specializations over time. It’s a time-consuming effort to maintain the general knowledge to manage IT well.

Mind you, that general knowledge is actually highly specialized tech knowledge. And very valuable it is too. So, in an effort to make it more accessible for myself, I’m gong to start writing down my thoughts. Maybe even organizing them.

And perhaps they will be useful not just for me. So here we go…