Wednesday, January 26, 2022
6075 Tagging
Monday, July 15, 2013
Googleversary
One year ago tomorrow I started my job as a Software Engineer for Google. It has been an amazing year so far, and I can't wait to get back to work tomorrow. I'm not supposed to reveal all the engineering trades secrets, but there are a few things that I can talk about that have really impressed me.
- One codebase. The vast majority of Google's engineers check their code into a single trunk. I am continually amazed that so many engineers can work together without stepping on each other's toes.
- Code reviews. Almost every single line of code that is committed to trunk is code-reviewed. While it's true that bugs do make it into the codebase, these code-reviews keep a consistent style while preventing errors and cross-functional issues.
- Craftsmanship. Everyone here seems to have it, and it is contagious. Even working with simple code, I find myself writing better, cleaner code.
- Engineering happiness. When you have this many engineers, every small improvement to the engineering process pays huge dividends. Because of this, Google spends some serious time an energy improving the engineering process.
This is just a quick sampling of what makes Google, Google. For more reading, there are some good explanations on the eng-tools blog (http://google-engtools.blogspot.com/)
Techstumbler started as a way for me to catalog all of the tech problems that I "stumbled" across, but couldn't find a good answer for elsewhere. Since joining Google, my output here has basically gone to zero for two reasons. First, while I was swimming in the deep-end for C#, .NET, Powershell in my previous job, I've had to strap on my swimmies and resign myself to shallower waters in the Java/Linux stack. Second, many, many of the tech problems I run into now relate to the Google infrastructure itself, which I can't really talk about, and wouldn't be useful to anyone not on that stack anyway.
All that said, I would like to start writing here again. I might try to post some thoughts about some of my other interests, but I'll try to keep things for the tech audience. We'll see what I come up with.
Tuesday, January 10, 2012
Loading Remote Assemblies in Powershell with .NET 4
We build our code to a central server and then use Powershell to install those build on our Development server. The upgrade to .NET 4 caused 2 problems with Powershell when we called
[Reflection.Assembly]::LoadFile()First:
Powershell by default runs in .NET 2.0. When we tried to load our new 4.0 assemblies, we got this error:
Exception calling "LoadFile" with "1" argument(s): "Could not load file or assembly 'file://\\buildServer\Application\assembly.dll' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded."
At C:\Scripts\Deployment\Deploy.ps1:XX char:XX
+ $assembly = [Reflection.Assembly]::LoadFile <<<< ($file); + CategoryInfo : NotSpecified: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : DotNetMethodException We need to tell Powershell to run in .NET 4.0 mode. To do that we need to create an app.config file for Powershell. In our case the file was created here: C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe.config, but YMMV based on Powershell's location. The new powershell.exe.config file looks like this:
<?xml version="1.0"?>
<configuration>
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedruntime version="v4.0.30319"/>
<supportedruntime version="v2.0.50727"/>
 </startup>
</configuration>
Now Powershell can run with both 2.0 and 4.0 assemblies.
Second:
The security features are a bit different when running 2.0 vs 4.0. This difference caused a problem when running assemblies from our Build server on our Development server:
Exception calling "LoadFile" with "1" argument(s): "An attempt was made to load an assembly from a network location which would have caused the assembly to be sandboxed in previous versions of the .NET Framework. This release of the .NET Framework does not enable CAS policy by default, so this load may be dangerous. If this load is not intended to sandbox the assembly, please enable the loadFromRemoteSources switch. See http://go.microsoft.com/fwlink/?LinkId=155569 for
more information."
At C:\Scripts\Deployment\Deploy.ps1:XX char:XX
+ $assembly = [Reflection.Assembly]::LoadFile <<<< ($file); + CategoryInfo : NotSpecified: (:) [], ParentContainsErrorRecordException + FullyQualifiedErrorId : DotNetMethodException Now we simply add the loadFromRemoteSources switch to the powershell.exe.config file we just created so that the entire file looks like this:
<?xml version="1.0"?>
<configuration>
<startup useLegacyV2RuntimeActivationPolicy="true">
<supportedruntime version="v4.0.30319"/>
<supportedruntime version="v2.0.50727"/>
</startup>
<runtime>
<loadfromremotesources enabled="true"/>
</runtime>
</configuration>
Now we are successfully loading remote .NET 4.0 assemblies in Powershell. Hope this helps.
Friday, August 26, 2011
Safari User Agent Detection in JavaScript
function detectSafari45()
{
var safariRegEx = /Mozilla\/5\.0 \([^\)]*\) AppleWebKit\/\d+\.\d+(\.\d+)? \(KHTML, like Gecko\) Version\/[45]\.\d+\.\d+( Mobile\/\w*)? Safari\/\d+\.\d+(\.\d+)?/i
var match = safariRegEx.exec(navigator.userAgent);
if (match !== null && match.length > 0)
{
return true;
}
return false;
}
Tuesday, May 10, 2011
Averaging Timespans in T-SQL
select
CONVERT(VARCHAR(13),creation_date,120) as [hour],
CONVERT(VARCHAR(8), max(stop_datetime - start_datetime), 108) as MaxWaitTime,
CONVERT(VARCHAR(8), min(stop_datetime - start_datetime), 108) as MinWaitTime,
CONVERT(VARCHAR(8), cast(avg(cast(stop_datetime - start_datetime as float)) as datetime), 108) as AvgWaitTime
from
[OpQueue].[Op]
where
start_datetime > '2011-05-07'
group by
CONVERT(VARCHAR(13),start_datetime ,120)
Here are a few things to note.
This statement will spit out the min, max, and average running time for operations in the queue that start in the same hour.
The third argument for convert is pretty handy for DATETIMEs. Here is the page where it is described: http://msdn.microsoft.com/en-us/library/ms187928.aspx.
The AVG() function doesn't work for DATETIMEs, so we need to convert it to a float and then back again to get what we are looking for.
Here's the graph we ended up with.
Seems like it's time to invest in some more processing power.
Friday, April 22, 2011
Running Hudson from OS X: the .war.zip Fiasco
java -jar hudson-2.0.0.war
to start up the service. I unzipped the file and ended up with a bunch of .class files and no .war. Ah, looks like I also unzipped the .war file. So then I found this nice post (http://superuser.com/questions/159260/in-mac-os-x-how-can-i-unzip-a-zip-file-without-unzipping-its-contents) on how to unzip a file without unzipping its contents. Now I have a
hudson-2.0.0.war/
directory. I ran
$ java -jar hudson-2.0.0.war
Invalid or corrupt jarfile hudson-2.0.0.war
Hmm, not sure what to do next. Somewhere in my search I read that a .war is basically just a zip file. So I tried
$ java -jar hudson-2.0.0.war.zip
on the original file, which worked. I renamed hudson-2.0.0.war.zip
$ mv hudson-2.0.0.war.zip hudson-2.0.0.war
and I was off and running.
So I hope this helps some other java n00bs who may have run into this. The moral of the story is .war == .zip
Friday, December 10, 2010
Moving Replicated FullText Index in SQL Server
SQL Server 2005
Transactional Replication with an Updateable Subscriber.
Here are the steps I took:
0. Make sure you have appropriate backups.
1. Point all of the apps to the Publisher because we will need to take the Subscriber offline.
2. On the Subscriber, open the synchronization status by right-clicking on Replication->Local Subscriptions->
3. Stop the Synchronization Service. (This really just pauses updates).
4. On the Subscriber run
SELECT name FROM sys.database_files WHERE type_desc = 'FULLTEXT'; to get the name of the FTI.5. On the Subscriber run
ALTER DATABASE [DB_NAME] SET OFFLINE;6. Move the FTI where to it's new home.
7. On the subscriber run
ALTER DATABASE [DB_NAME] MODIFY FILE (Name=[FTI_NAME], Filename = "'new/location/on/disk/'); 8. On the Subscriber run
ALTER DATABASE [DB_NAME] SET ONLINE;9. Start the Sync Service in View Synchronization Status.
That should be it. Hope this helps.
Wednesday, November 24, 2010
T-SQL Query: Tables Without a Primary Keys
select s.name+'.'+ t.name
from
sys.schemas s
join sys.tables t on s.schema_id = t.schema_id
left join sys.indexes i on i.object_id = t.object_id and
i.is_primary_key = 1
where i.name is null
order by (s.name+'.'+ t.name) asc;
Thursday, May 27, 2010
Great Post on SQL Server Stored Procedure Variables
TOP clause of a MSSQL stored prodedure. This post answered all my questions:http://sqlserver2000.databases.aspfaq.com/how-do-i-use-a-variable-in-a-top-clause-in-sql-server.html
Here's the best part
CREATE PROCEDURE dbo.getFoo
@top INT AS
BEGIN
SET ROWCOUNT @top
SELECT foo
FROM blat
ORDER BY foo DESC
-- never forget to set it back to 0!
SET ROWCOUNT 0
END
GO
Thursday, March 25, 2010
Programmaticly Changing the File Attribute of an App.config File.
public static bool SetEnvironment(string env)
{
string configFilePath = string.Empty;
env = env.ToLowerInvariant();
switch (env)
{
case "dev":
configFilePath = @"Config\Dev.config";
break;
case "prod":
configFilePath = @"Config\Prod.config";
break;
default:
return false;
}
try
{
System.Configuration.Configuration config =
ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None);
AppSettingsSection appSetSec = config.AppSettings;
appSetSec.File = configFilePath;
config.Save(ConfigurationSaveMode.Modified);
ConfigurationManager.RefreshSection("appSettings");
}
catch (Exception ex)
{
OfflineDemoTool.WriteError(ex.Message);
return false;
}
return true;
}
One thing that I forgot to do was to make sure that the Config\Dev.config and Config\Prod.config files were set to "Copy if newer" in their properties list, otherwise they won't be copied to the output directory.
Most of this comes straight from the MSDN article found here: http://msdn.microsoft.com/en-us/library/system.configuration.appsettingssection.aspx
Wednesday, December 9, 2009
Windows Commands with Arguments in Powershell
[string]$pathToExe = "C:\MySQL\MySQL Server 5.1\bin\mysqldump.exe";
[string]$user = "myUser";
[string]$password = "myPass";
[string]$dbName = "myDB";
[Array]$arguments = "-u", $user, "--password=$password", $dbName;
& $pathToExe $arguments | Out-File -FileName "out.sql";
So there you have it. That's how to run an executable with spaces and arguments from PowerShell.
While this article from PowerShell.com didn't answer my questions, I thought it was pretty useful and relevent:
http://powershell.com/cs/blogs/ebook/archive/2009/03/30/chapter-12-command-discovery-and-scriptblocks.aspx
Tuesday, November 10, 2009
Changing aspnet_wp user in IIS 5.1
We needed the asp.net worker process (aspnet_wp.exe) to access a share on a remote machine. By default the "Netowrk Service" runs aspnet_wp and that user doesn't have access to remote shares. The way to change this is by changing the "processModel" attribute in your machine.config file (c:\windows\microsoft.net\framework\v2*\config\machine.config).
Mine ended up looking something like this:
<system.web>
<processModel userName="DOMAIN\username" password="password" autoConfig="true">
</processModel>
...
</system.web>
Just make sure to restart IIS for the changes to take effect.
Tuesday, May 19, 2009
Powershell Diff Directory
$fromDir = "C:\FromDir";
$toDir = "C:\ToDir";
$diffDir = "C:\DiffDir";
Get-ChildItem -Recurse -Path $fromDir | % {
if ((Test-Path (Join-Path -Path $toDir -ChildPath ([string]$_.FullName).Replace($fromDir, ""))) -eq $false)
{
if ((Test-Path (Join-Path -Path $diffDir -ChildPath ([string]$_.Directory).Replace($fromDir, ""))) -eq $false)
{
New-Item -Type "Directory" -Path (Join-Path -Path $diffDir -ChildPath ([string]$_.Directory).Replace($fromDir, ""));
}
Copy-Item -Recurse -Force $_.FullName -Destination (Join-Path -Path $diffDir -ChildPath ([string]$_.FullName).Replace($fromDir, ""));
}
};
As always, if anyone know a better way to do this, let me know.
Wednesday, April 29, 2009
Powershell Rename-Item vs Copy-Item: Changing file names to upper case
Get-ChildItem -Recurse -Path "C:\media" | % {Rename-Item $_.FullName -NewName ([string]$_.Name).ToUpper()};
However, I received the following error:
Rename-Item : Source and destination path must be different.
So it looks like PowerShell refused to rename
file1 to FILE1, because it thinks that they are the same file. In order to get around this I ended up copying the entire tree, and even that command isn't particularly elegant:
Get-ChildItem -Recurse -Path "C:\media" | % {Copy-Item -Force $_.FullName ([string]$_.FullName).Replace("C:\media", "C:\MEDIA2").ToUpper()};
If there is a better way to do this, I'd love to know.
Thursday, September 18, 2008
XPlanner Authentication with NTLM
The XPlanner install was happening on the lone linux box (Fedora 8) on an NT domain. The box was almost completely empty so I had to install java first. I chose to install the jsdk 1.4.2 even though that version's is EOL'd. XPlanner hasn't been updated in a while (since 2006) and I didn't want to mess w/ a new version of the sdk.
I upacked and installed the "standalone" version of XPlanner 0.7b7. So far so good. I was immediately able to use the default username/password to login.
The hardest part was figuring out how to authenticate based on our NT credentials. Most of the configuration occurs in the xplanner-0.7b7-standalone/webapps/ROOT/WEB-INF/classes/xplanner.properties file.
Here's how I was able to get the NT authentication to work. In the xplanner.properties file, find the authentication strings:
#
# XPlanner security configuration
#
xplanner.security.login[0].module=com.technoetic.xplanner.security.module.XPlannerLoginModule
xplanner.security.login[0].name=XPlanner
xplanner.security.login[0].option.userIdCaseSensitive=true
xplanner.security.login[0].option.debug=true
#xplanner.security.login[1].module=com.technoetic.xplanner.security.module.jndi.JNDILoginModule
#xplanner.security.login[1].name=JNDI
#xplanner.security.login[1].option.userIdCaseSensitive=false
#xplanner.security.login[1].option.debug=true
#xplanner.security.login[1].option.connectionURL=
#xplanner.security.login[1].option.connectionName=cn=
#xplanner.security.login[1].option.connectionPassword=
#xplanner.security.login[1].option.digest=SHA
#xplanner.security.login[1].option.userPattern=
#xplanner.security.login[1].option.userPassword=
#xplanner.security.login[1].option.authentication=simple
#xplanner.security.login[1].option.derefAliases=never
#xplanner.security.login[1].option.userSearch=cn={0}
#xplanner.security.login[1].option.userSubtree=true
#xplanner.security.login[1].option.roleBase=
#xplanner.security.login[1].option.roleName=
#xplanner.security.login[1].option.roleSearch=(uniqueMember={0})
# NTLM login module
#xplanner.security.login[2].module=com.technoetic.xplanner.security.module.ntlm.NtlmLoginModule
#xplanner.security.login[2].name=NTLM
#xplanner.security.login[2].option.userIdCaseSensitive=false
#xplanner.security.login[2].option.domain=DOMAIN
#xplanner.security.login[2].option.controller=CONTROLLER
To use the NTLM module, comment out the first set to security strings, uncomment the NTLM strings, and change the NTLM array index to 0. Here's how my file looked after I finished.
#
# XPlanner security configuration
#
#xplanner.security.login[0].module=com.technoetic.xplanner.security.module.XPlannerLoginModule
#xplanner.security.login[0].name=XPlanner
#xplanner.security.login[0].option.userIdCaseSensitive=true
#xplanner.security.login[0].option.debug=true
#xplanner.security.login[1].module=com.technoetic.xplanner.security.module.jndi.JNDILoginModule
#xplanner.security.login[1].name=JNDI
#xplanner.security.login[1].option.userIdCaseSensitive=false
#xplanner.security.login[1].option.debug=true
#xplanner.security.login[1].option.connectionURL=
#xplanner.security.login[1].option.connectionName=cn=
#xplanner.security.login[1].option.connectionPassword=
#xplanner.security.login[1].option.digest=SHA
#xplanner.security.login[1].option.userPattern=
#xplanner.security.login[1].option.userPassword=
#xplanner.security.login[1].option.authentication=simple
#xplanner.security.login[1].option.derefAliases=never
#xplanner.security.login[1].option.userSearch=cn={0}
#xplanner.security.login[1].option.userSubtree=true
#xplanner.security.login[1].option.roleBase=
#xplanner.security.login[1].option.roleName=
#xplanner.security.login[1].option.roleSearch=(uniqueMember={0})
# NTLM login module
xplanner.security.login[0].module=com.technoetic.xplanner.security.module.ntlm.NtlmLoginModule
xplanner.security.login[0].name=NTLM
xplanner.security.login[0].option.userIdCaseSensitive=false
xplanner.security.login[0].option.domain=DOMAIN
xplanner.security.login[0].option.controller=CONTROLLER
The NTLM module uses the local DB as a fall back, so any logins that you creaed locally that aren't in the Active Directory should still work.
Tuesday, September 9, 2008
JPEG Marker codes
Tuesday, June 3, 2008
eApps and yum
PHP Warning: Module 'modulename' already loaded in Unknown on line 0As it turns out that means that the xls package was included in the binary, and also being declared as a dynamic package. So, to fix this I commented out the following line in the /etc/php.d/xsl.ini:
;extension=xsl.soNow on to yum itself. Doing any kind of yum update resulted in the following error:
Error: Missing Dependency: glibc-common = 2.3.4-2.36 is needed by package glibc-dummy-centos-4
After doing a bit of research, I found that the dummy-centos-4 package isn't really necessary if you're going to be upgrading your gcc libraries anyways so away it goes.
yum remove glibc-dummy-centos-4After that I ran a full yum upgrade which worked just fine.
Friday, May 2, 2008
Stripping Accennts From Text With PHP
$text = iconv('UTF-8', 'ASCII//TRANSLIT', $text);Done and done.
Monday, April 14, 2008
SQL Join with count
SELECT
COUNT(listings.id), offices.id
FROM offices LEFT JOIN listings ON (offices.id = listings.office_id)
WHERE
listings.status = 1
However, this only returned office/count pair where the count was greater than 0. It seemed like the WHERE condition wasn't being applied correctly, like it was being applied AFTER the tables were joined. I wanted the WHERE condition to be applied to the listings table and then have the results joined to the offices table. So, here's the solution I found: in situations like this, the WHERE clause needed to be added to the ON clause in the LEFT JOIN like this:
SELECT
COUNT(listings.id), offices.id
FROM offices LEFT JOIN listings ON (offices.id = listings.office_id AND listings.status = 1)
Works like a charm.
