Post #22 AWS Outage. My Domain's First

Nothing is perfect. All promises are hollow and if you get a free service you can tell. Yesterday I faced my first domain outage. It was intermittent. When I got to my AWS console I saw my instance state was in error. I had no clue what to do. I tried after some time and the site worked but after navigating a couple of pages it stopped again.

I ended up on Elastic Bean Stalk and decided to restart the instance but there was no such option. I had a reboot option but it did not allow me to reboot. There were 2 other options Stop and Start and so I tried stopping it. It gave an error saying the instance is in no shape to accept my command. I tried accessing again and the intermittent failure continued.

After a while when I reached back on EC2 screen, the instance was shutting down and a brand new instance was available. At this point I did not know what to do. Do I reconfigure the entire domain again. In fact I had forgotten it because I did that in steps over days. That sent shudders up my spine.  I tried ssh using putty but that did not go through either.

It was getting late and I decided to go to sleep. The next morning, I figured that the old instance had completely disappeared. The Load Balancer and other stuff was reconfigured and the site was working but the issue continued.

Also, all my logs and data was missing on the new server. I created a new directory for temp statements. The intermittent nature of the problem however continued.

I checked at all the places. The EC 2 instance, Route 53, Elastic Beanstalk, so on. But could not identify the problem. Later I saw an error with the load balancer. I think the load balancer itself balances the load between 4 instances in the same region and 3 of 4 said not available. I just removed these from there and it worked.

I think I need to document how to manage the site using AWS. Also there are no great forums which help you trouble shooting and what all to check etc. I tried logging a ticket but it did not give me an option saying you need to buy that option.

The question is how do you support potential customers. The fact that I created a site and hosted it there should be a strategy to retain me even if its free trial period so that the customer can feel comfortable. Currently, I do not feel confident about my long term plans to host it on AWS. The support sucks...

Post #22 Basic Stuff

This weekend I wrapped a few less important usecases.
- Viewing your transaction upload history
- Updating the FAQ section, making it logically complete by writing about all important and complex use cases.
- Cleaning up the navigation a bit.

I forgot to mention these in my rant on config mgt.

I'm now thinking about moving to mobile devices. Let me invest the next month or so building a mobile device interface using sencha.

The email part, I'll visit later. I'll have to invest money and but email hosting before I can make use of it. I do not want to send emails from gmail...

The footer content and the T&C still remain unfinished. The work is truly never finished. There is always something to add or improve.

Retirement Planning Revisited

I'm facinated by retirement. If I had enough money I'd retire tomorrow. By retire, I mean not working in the corporate sector just to make my ends meet. I'll splurge money on my pet projects that no one but me will benefit from. Alas! I'm not there yet and don't seem to be getting there anytime soon either.

About 5 years ago I blogged about retirement planning. I wrote about projecting your retirement money remaining at the end of any year. Just to refresh here is the formula

i = Rate of return on your investment (post tax)
i_f = Inflation
P₀ = Initial Savings
B = Annual expenses (burn rate)
R = (1 + i_f )/(1+ i)
n = Time in years

Money remaining at the end of n^th year

P_n = P₀ X (1+i)ⁿ – B X (1+i)^n-1 X (Rⁿ-1)/(R-1)

Its another one of those useless things that no one but I found any value in. But over the years as I fantasized retirement, I noticed an interesting trend. 

What I noticed was that in most cases I'd finally run out of money. Dh!

But then if I save till a point I'm calling "inflection point" everything will change. 

The inflection point is the initial amount P₀ you being with and if the other variables are held constant, namely inflation, interest you earn on your savings, you'll never ever run out of money. 

Not in 100 years and not in 1000 years. Never. Just like escape velocity, you'll never return to earth.

That value can be derived from the above equation with n-> Infinity. Assuming i_f(inflation) is lower than i(interest earned) of course. 

Calculating the dream P₀ 

P₀ = B/(R-1)(1+i) 

gives you the initial amount that will NEVER run out....

An interesting point :)

Post #21 Configuration Management Nightmare

So I galloped to a greatly productive this weekend and by Sunday evening was ready for another deployment till I ran into a nightmare called poor configuration management.

I've always known the importance of it. So many releases / deployments have been spoiled by this innocuous looking thing. The importance of it grows with the team size. In a distributed environment teams use concepts like continuous integration and weekly / daily merges to avoid late surprises. And yet it sneaks in from time to time and ruins an otherwise perfectly successful release.

I understand this issue and though I'm the only one working on this project I'd decided to practice it.

Till nov-12, I was doing only development and therefore I had just one environment. I made builds several times a day and was fine with it. In November, I went live on AWS. There were some environment variables that were different for production environment and I created 2 projects instead. One that mirrored the production and the other that I used for development. I'd typically remember the files I needed to promote and I'd do that once I reached a logical milestone. I did not promote these changes to production immediately though. I would do it about twice a month. Also typically, I upload my statements once a month which meant that a large part of the functionality was unused for most part of the month. This approach worked well till Jan, but as I tried uploading Jan statements, I ran into my first brush with config mgt.

I typically store the uploaded statements in a temp folder which I purge from time to time. The location of the folder is different on the production env. Also, its hardcoded in one of the java files ( A decidedly poor practice). In Jan I was working on some functionality and updated this file. I later promoted this file to production. This obviously overwrote the path to temp folder. As I did not use the functionality till Feb, I did not run into any problem. When I did, it took me a while to figure out what was wrong. Why my statements were not processed?

And so I learnt a lesson and after some research, decided to go with Visual SVN for repository. I got eclipse client to connect to it and set up both production and local as 2 projects.

That helped me in identifying which files to promote and I do not need to remember file names as before.
But, there is a step in between to compare the new version with the HEAD version to ensure the changes are fine and do not break anything.

Yesterday, I missed that step and paid the price. The error happened right at the time you keyed in DomainName.com. It'd show up an ugly 500 message with the entire stack trace. Yes, that's Ugly. Also, the stack trace was not much use.

I had made some changes to a file to manage guest user and I thought that might be the root cause. But after several attempts to correct the problem it still did not go away. This brings me to the effectiveness of logging.

I was always proactive about logging. I have avoided writing elaborate Try Catch and have use AOP based class to intercept and log the errors. This does a decent job of managing errors on the business tier. But when it comes to framework, if there was an error in the framework the whole thing falls apart. Especially spring errors etc.

Anyhow, after much trouble I found out that I had unwittingly promoted changes meant for mobile  devices to production and hence the error.

Net Net 2 lessons.
- Get Config Mgt right.
- Improve Error Logging

Post #20 Split Adjustments

Wrapped up Split Adjustments. Its time now to focus on Mobile. There is some static content to create as well. The FAQ section is still incomplete. Terms of Use / Privacy Policy, About, Contact Us and Feedback links are pending.

Boss wants to talk about my career growth. Sends shivers up my spine. Boss calling to discuss your growth means your time is up...

Post #19 Profile Done

Of the list a few more are ticked off.

I also need some static content such as About us(me). Terms and conditions. Thats a must do item. I do not know how to do it without giving my identity away.Then Contact us etc. I also need to update the FAQ. Have done some good work in the direction but still a long way to go.

- Remember me 
- Redirecting to requested resource after log in instead of standard landing page

- Redirecting remember me user to a standard page
- Redirecting based on device (mobile vs desktop)

• - I've found out how to do it, using Spring. The question is how to handle it. I could for example redirect to a subdomain or I could redirect to a url such as /m. Neither of these work for me as a lot of my calls are ajax and if I change the url etc. I'm not able to process the request using the same controller method. Spring says I can handle multiple URLs using same controller method, but its not working for me. 

I'll have to find a better way to make it work for my site.

- Cash Txn Widget
- Adding Date to Cash Txn
- Distinction between one  time cash txn vs recurring cash txn
- Change Pwd/Forget Password
- Email notifications

• eEmail. The problem is with AWS not allowing me to send emails from the domain name. I do not have an active email account for my domain. I'll have to but email service. Right now, I'm not in a mood to splurge a lot of money. This will have to wait.  
• Another option is to send out emails using my gmail account or create a temporary account on gmail to send out emails. This will work but is inelegent. 

- View Split Transactions

• I'll finish this off by end of the Feb. This is useful. I also had to fix a this functionality. When loading statements, the algorithm was unable to figure out split txns and hence loading duplicate copies. Took a long time to fix it but now its done. View Split Txn is the next logical step.

- Elegant impl of labels vs the rest
- Alerts
- Mobile interface to capture cash txn
- Mobile interface to view performance against budgets
- Mobile interface to view savings
- Enhance Help
- Redo the Navigation

Up Next: 
- View Split Txn
- Redirection based on device
- Change PWD/Forget Pwd
- Email notifications
- Profile Mgt (new one)

Domain Goes Live!

I thought about it a lot. Why should I spend money on something that only I use and is going to be free atleast for the next 6 months anyways through AWS . Also, not sure how will my org take it. They had no role in building it. This is surely not a business for me. Just an indulgence.It keeps me going and helps me learn...

I already had the name in my mind. I had checked about a year back and it was available. I just checked it again and it showed up as available. Alongside there was a note saying "Snap it up before someone else does". It did not cost an arm or a leg and so I thought and then thought some more and then just did it. I just could not resist. I lapped it up. Damages: Rs.3300 for 5 years. That was for both .com and .in together.

Thats when the struggle started.From the domain registrar how do I point the domain to my AWS instance. Learnt a new concept. Researched a lot, saw a few videos and finally completed it an hour ago.

So, my toy is live and has a name too.