Evaluating New Technology

Four principles for evaluating new technology:

  1. Does it have a practical use case.
  2. Is the technology’s performance sufficient to meet that use case.
  3. Does it exceed competing and already existing products at that use case.
  4. Does the use case apply to a sufficient number of users to support the size of ecosystem needed for the technology to be viable.

Workflows For Developers

As you become more experienced as a software developer, much of improving your skill set becomes perfecting ‘workflows’. This concept is very common to professionals in other areas like graphics design, but I see it talked about much less often in software engineering. Viewing development from this perspective requires that you look at the multiple technologies, approaches, frameworks, tools, libraries, etc… as parts of an overall system. A properly constructed workflow should have ‘default’ solutions that answer the question “how do I handle this situation 90% of the time?”. Good workflows will minimize the time wasted considering options to implement ‘easy stuff’ and will make more time to focus on the real challenges and to build additional value into your solutions.

Implementing a workflow does not mean that you have to fully automate development or that you suck out all of the creativity or intelligence from the process. The value of a ‘good’ software engineer comes from the solutions you can provide to the 10% of system construction that does not fall within a standard workflow. But if you find yourself frequently having to stop and think about the various ways you can do something you probably don’t have your workflow sufficiently developed.

On the downside, coming up with a workflow that fits your approach to development will require a lot of time and effort minimizing all that you know to a core set. Here are some practical tips for speeding up the process:

1. Decide on a core set of development tools, libraries and frameworks. These tools should ideally incorporate a particular philosophy to solving problems that you should follow most of the time (for instance, using Angular and its approach to UI, only falling back to something like direct jQuery development when you need to do something Angular can’t handle). Once your pick a framework that matches your development preferences, resist the urge to stray away from the tool’s ‘standard approach’. Stepping out of your framework should be the exception.

2. Read books on patterns… all kinds of patterns. Avoid recreating the wheel.

Examples:

Analysis patterns: http://www.amazon.com/Object-Oriented-Analysis-Design-Applications-Edition/dp/020189551X/ref=sr_1_1?ie=UTF8&qid=1396199795&sr=8-1&keywords=object+analysis+and+design+booch

Database patterns: http://www.amazon.com/Data-Model-Resource-Book-Vol/dp/0470178450/ref=sr_1_3?ie=UTF8&qid=1396199857&sr=8-3&keywords=database+design+patterns

UI Patterns: http://www.amazon.com/Mobile-Design-Pattern-Gallery-Applications/dp/1449314325/ref=sr_1_2?ie=UTF8&qid=1396199881&sr=8-2&keywords=UI+patterns

3. Avoid “the new hotness”. You can evaluate new technologies as they mature to consider adopting them into your standard process, but modifying your workflow is time consuming and expensive and should be treated as a major step.

4. Minimize the time you spend considering options when you come to a point where you are not sure what solution is best. Remember, “There are no wrong or right answers, just more or less useful solutions”. It is sometimes better to make a decision and go with it than to spend excessive amounts of time agonizing over the options.

Multiplication Aggregate Function in Microsoft SQL

Microsoft SQL has many useful aggregate functions such as AVG, SUM, COUNT — but it doesn’t have a an aggregate function for multiplication. This type of aggregate function would be very useful for needs such as calculating a geometric average.

With a little math, it is possible to calculate a multiplication aggregate result. The following line of SQL is an example of an application of this technique for a field that contains percentages:

SELECT (EXP(SUM(LOG(COALESCE(1 + ([FIELD NAME]/100),1)))) – 1) * 100

The approach is not as efficient as a built-in function, but it may offer superior performance to alternatives.

Quick Tips for Form Submit Security

I never cease to be amazed at the number of supposedly ‘experienced developers’ I encounter that are not familiar with even the basic requirements of building a secure application. I am far from a security expert, but every developer building Web applications should take steps to familiarize themselves with application security.

As an introduction, I’ve jotted down a few tips for securely handling form submits. This obviously doesn’t cover everything (consider this the MINIMUM you should be doing) but these guidelines should give you a good start.

1. Force types on user submitted values. If the field is a number, convert it to a number and only allow valid number values.

2. Limit length on user submitted string values as much as possible.

3. Parse user submitted string values for dangerous characters. Some languages have functions built in for this. If your language doesn’t have built in methods, find a good library like the OWASP Java Encoder Project.

4. Use your language’s version of ‘parameterized  queries’ or ‘prepared statements’. Don’t EVER append a user submitted string to a SQL statement through string concatenation.

5. Use password hashing with a current hashing algorithm. MD5 and SHA-1 aren’t good enough. Use a method like bcrypt and properly use salting and stretching.

6. Be careful when displaying information based on ID’s or key values passed in through POST or GET parameters. It’s very simple to change these values and trick your system into revealing information that a user should not have access to.

7. NEVER EVER rely on client-side Javascript to provide any form of security for your application. You would think this would be obvious, but I have been called in to consult on several applications where this was done. (really, don’t do this…)

8. Bookmark this site: OWASP. This is one of the best resources for learning to build secure web applications. I’ve been following it for almost a decade. Warning: It can be overwhelming to those just starting out with application security. Just take it a step at a time.

Final Tip: Try to use libraries for security functions (from respectable sources of course …I suppose Microsoft counts as a respectable source). I’ve been working with application security for 15 years and I am still don’t consider myself expert enough to attempt to write my own security related code. The ‘bad guys’ are really, really smart and it takes a great deal of expertise to write systems to protect against their constantly evolving attacks.

Linear Regression With Dates

Most of the math libraries I use don’t support date values for linear regression. This was my work around.

1. Convert date / times to Unix time.

2. Take the earliest date and subtract that time value from all the other values.
3. Run linear regression using the modified Unix times.
4. Add the earliest time value back to each Unix time when stating results.

Simple Date Table Script: MS SQL

This script creates a date table for Microsoft SQL 2008 and higher to be used for joins in certain types of date related queries. The script was based on this post but I made several changes including adding ISO week and ISO year fields, converting the key to a Unix Time calculation, and adding an “end of month” flag. The script is not particularly efficient, but it is only intended to be run once so that was not a concern.

Table Structure

CREATE TABLE [dbo].[DateLookup](
[UnixTime] [bigint] NOT NULL,
[DateFull] [datetime] NULL,
[USRegionFormat] [varchar](10) NULL,
[FullYear] [char](4) NULL,
[QuarterNumber] [tinyint] NULL,
[WeekNumber] [tinyint] NULL,
[WeekDayName] [varchar](10) NULL,
[MonthDay] [tinyint] NULL,
[YearDay] [smallint] NULL,
[WeekDay] [tinyint] NULL,
[MonthNumber] [tinyint] NULL,
[ISOYear] [smallint] NULL,
[ISOWeek] [tinyint] NULL,
[MonthName] [nvarchar](12) NULL,
[EndOfMonth] [bit] NULL,
PRIMARY KEY CLUSTERED
(
[UnixTime] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]

Script

DECLARE @Date DATETIME
SET @Date = '1/1/1990'

WHILE @Date < '1/1/2100'
BEGIN
INSERT INTO DateLookup
(
UnixTime,
DateFull,
FullYear,
QuarterNumber,
WeekNumber,
WeekDayName,
MonthDay,
YearDay,
USRegionFormat,
[WeekDay],
MonthNumber,
ISOWeek,
ISOYear,
[MonthName],
EndOfMonth
)
SELECT
CAST(DATEDIFF(MINUTE,{d '1970-01-01'}, @Date) AS bigint) * 60, --UnixTime
@Date, -- DateFull
YEAR(@Date), -- FullYear
DATEPART(qq, @Date), --QuarterNumber
DATEPART(ww, @Date), --WeekNumber
DATENAME(dw, @Date), --WeekDayName
DATEPART(dd, @Date), --MonthDay
DATEPART(dy, @Date), --YearDay
CONVERT(VARCHAR(10), @Date, 101), --CharacterDate
DATEPART(dw, @Date), --[WeekDay]
DATEPART(mm, @Date), --MonthNumber

DATEPART(ISO_WEEK,@Date), --ISOWeek

CASE --ISOYear
WHEN DATEPART(ISO_WEEK, @Date) = 1 and DATEPART(MONTH, @Date) = 12
THEN DATEPART(year, @Date) + 1
WHEN DATEPART(ISO_WEEK, @Date) = 53 and DATEPART(MONTH, @Date) = 1
THEN DATEPART(year, @Date) -1
ELSE DATEPART(year, @Date)
END,

DATENAME(month, @Date), --MonthName

CASE --EndOfMonth
WHEN CONVERT(VARCHAR(10), DATEADD(s,-1,DATEADD(mm, DATEDIFF(m,0,@Date)+1,0)), 101) = CONVERT(VARCHAR(10), @Date, 101)
THEN 1
ELSE 0
END

SET @Date = DATEADD(dd, 1, @Date)
END